{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.6"
    },
    "colab": {
      "name": "1st_CS-LunarLander-dqn.ipynb",
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "oBBH0k8rc5iW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e2d9a444-e367-4d80-f120-a3aaf4ea1fc7"
      },
      "source": [
        "%clear all\n",
        "\n",
        "import typing\n",
        "\n",
        "import torch\n",
        "from torch import nn\n",
        "\n",
        "\n",
        "QNetwork = nn.Module\n",
        "\n",
        "\n",
        "class LambdaLayer(nn.Module):\n",
        "    \n",
        "    def __init__(self, f):\n",
        "        super().__init__()\n",
        "        self._f = f\n",
        "        \n",
        "    def forward(self, X):\n",
        "        return self._f(X)\n",
        "\n",
        "\n",
        "def make_deep_q_network_fn(action_size: int) -> typing.Callable[[], QNetwork]:\n",
        "    \n",
        "    def deep_q_network_fn() -> QNetwork:\n",
        "        q_network = nn.Sequential(\n",
        "            nn.Conv2d(in_channels=4, out_channels=32, kernel_size=8, stride=4),\n",
        "            nn.ReLU(),\n",
        "            nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2),\n",
        "            nn.ReLU(),\n",
        "            nn.Conv2d(in_channels=64, out_channels=64, kernel_size=2, stride=1),\n",
        "            nn.ReLU(),\n",
        "            LambdaLayer(lambda tensor: tensor.view(tensor.size(0), -1)),\n",
        "            nn.Linear(in_features=25024, out_features=512),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(in_features=512, out_features=2)\n",
        "        )\n",
        "        return q_network\n",
        "    \n",
        "    return deep_q_network_fn"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[H\u001b[2J"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e9fDs-Vwc5ii",
        "colab_type": "text"
      },
      "source": [
        "### The Loss Function\n",
        "\n",
        "The $Q$-learning update at iteration $i$ uses the following loss function\n",
        "\n",
        "$$ \\mathcal{L_i}(\\theta_i) = \\mathbb{E}_{(s, a, r, s') \\sim U(D)} \\Bigg[\\bigg(r + \\gamma \\max_{a'} Q\\big(s', a'; \\theta_i^{-}\\big) - Q\\big(s, a; \\theta_i\\big)\\bigg)^2\\Bigg] $$\n",
        "\n",
        "where $\\gamma$ is the discount factor determining the agent’s horizon, $\\theta_i$ are the parameters of the $Q$-network at iteration $i$ and $\\theta_i^{-}$ are the $Q$-network parameters used to compute the target at iteration $i$. The target network parameters $\\theta_i^{-}$ are only updated with the $Q$-network parameters $\\theta_i$ every $C$ steps and are held fixed between individual updates. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bCWYziRYc5ij",
        "colab_type": "text"
      },
      "source": [
        "### Experience Replay\n",
        "\n",
        "To perform *experience replay* the authors store the agent's experiences $e_t$ as represented by the tuple\n",
        "\n",
        "$$ e_t = (s_t, a_t, r_t, s_{t+1}) $$\n",
        "\n",
        "consisting of the observed state in period $t$, the reward received in period $t$, the action taken in period $t$, and the resulting state in period $t+1$. The dataset of agent experiences at period $t$ consists of the set of past experiences.\n",
        "\n",
        "$$ D_t = \\{e1, e2, ..., e_t \\} $$\n",
        "\n",
        "Depending on the task it may note be feasible for the agent to store the entire history of past experiences.\n",
        "\n",
        "During learning Q-learning updates are computed based on samples (or minibatches) of experience $(s,a,r,s')$, drawn uniformly at random from the pool of stored samples $D_t$.\n",
        "\n",
        "The following is my Python implmentation of these ideas. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Of1-f8Adc5il",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import collections\n",
        "import typing\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "\n",
        "_field_names = [\n",
        "    \"state\",\n",
        "    \"action\",\n",
        "    \"reward\",\n",
        "    \"next_state\",\n",
        "    \"done\"\n",
        "]\n",
        "Experience = collections.namedtuple(\"Experience\", field_names=_field_names)\n",
        "\n",
        "\n",
        "class ExperienceReplayBuffer:\n",
        "    \"\"\"Fixed-size buffer to store experience tuples.\"\"\"\n",
        "\n",
        "    def __init__(self,\n",
        "                 batch_size: int,\n",
        "                 buffer_size: int = None,\n",
        "                 random_state: np.random.RandomState = None) -> None:\n",
        "        \"\"\"\n",
        "        Initialize an ExperienceReplayBuffer object.\n",
        "\n",
        "        Parameters:\n",
        "        -----------\n",
        "        buffer_size (int): maximum size of buffer\n",
        "        batch_size (int): size of each training batch\n",
        "        seed (int): random seed\n",
        "        \n",
        "        \"\"\"\n",
        "        self._batch_size = batch_size\n",
        "        self._buffer_size = buffer_size\n",
        "        self._buffer = collections.deque(maxlen=buffer_size)\n",
        "        self._random_state = np.random.RandomState() if random_state is None else random_state\n",
        "        \n",
        "    def __len__(self) -> int:\n",
        "        return len(self._buffer)\n",
        "    \n",
        "    @property\n",
        "    def batch_size(self) -> int:\n",
        "        return self._batch_size\n",
        "    \n",
        "    @property\n",
        "    def buffer_size(self) -> int:\n",
        "        return self._buffer_size\n",
        "\n",
        "    def is_full(self) -> bool:\n",
        "        return len(self._buffer) == self._buffer_size\n",
        "    \n",
        "    def append(self, experience: Experience) -> None:\n",
        "        \"\"\"Add a new experience to memory.\"\"\"\n",
        "        self._buffer.append(experience)\n",
        "    \n",
        "    def sample(self) -> typing.List[Experience]:\n",
        "        \"\"\"Randomly sample a batch of experiences from memory.\"\"\"\n",
        "        idxs = self._random_state.randint(len(self._buffer), size=self._batch_size)\n",
        "        experiences = [self._buffer[idx] for idx in idxs]\n",
        "        return experiences\n"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v5iAtJXjc5i6",
        "colab_type": "text"
      },
      "source": [
        "### The Deep Q-Network Algorithm\n",
        "\n",
        "The following is Python pseudo-code for the Deep Q-Network (DQN) algorithm. For more fine-grained details of the DQN algorithm see the methods section of [Minh et al 2015](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf).\n",
        "\n",
        "```python\n",
        "\n",
        "# hyper-parameters\n",
        "batch_size = 32 # number of experience tuples used in computing the gradient descent parameter update.\n",
        "buffer_size = 10000 # number of experience tuples stored in the replay buffer\n",
        "gamma = 0.99 # discount factor used in the Q-learning update\n",
        "target_network_update_frequency = 4 # frequency (measured in parameter updates) with which target network is updated.\n",
        "update_frequency = 4 # frequency (measured in number of timesteps) with which q-network parameters are updated.\n",
        "\n",
        "# initilizing the various data structures\n",
        "replay_buffer = ExperienceReplayBuffer(batch_size, buffer_size, seed)\n",
        "local_q_network = initialize_q_network()\n",
        "target_q_network = initialize_q_network()\n",
        "synchronize_q_networks(target_q_network, local_q_network)\n",
        "\n",
        "for i in range(number_episodes)\n",
        "\n",
        "    # initialize the environment state\n",
        "    state = env.reset()\n",
        "\n",
        "    # simulate a single training episode\n",
        "    done = False\n",
        "    timesteps = 0\n",
        "    parameter_updates = 0\n",
        "    while not done:\n",
        "\n",
        "        # greedy action based on Q(s, a; theta)\n",
        "        action = agent.choose_epsilon_greedy_action(state) \n",
        "\n",
        "        # update the environment based on the chosen action\n",
        "        next_state, reward, done = env.step(action)\n",
        "\n",
        "        # agent records experience in its replay buffer\n",
        "        experience = (state, action, reward, next_state, done)\n",
        "        agent.replay_buffer.append(experience)\n",
        "\n",
        "        # agent samples a mini-batch of experiences from its replay buffer\n",
        "        experiences = agent.replay_buffer.sample()\n",
        "        states, actions, rewards, next_states, dones = experiences\n",
        "\n",
        "        # agent learns every update_frequency timesteps\n",
        "        if timesteps % update_frequency == 0:\n",
        "            \n",
        "            # compute the Q^- values using the Q-learning formula\n",
        "            target_q_values = q_learning_update(target_q_network, rewards, next_states, dones)\n",
        "\n",
        "            # compute the Q values\n",
        "            local_q_values = local_q_network(states, actions)\n",
        "\n",
        "            # agent updates the parameters theta using gradient descent\n",
        "            loss = mean_squared_error(target_q_values, local_q_values)\n",
        "            gradient_descent_update(loss)\n",
        "            \n",
        "            parameter_updates += 1\n",
        "\n",
        "        # every target_network_update_frequency timesteps set theta^- = theta\n",
        "        if parameter_updates % target_network_update_frequency == 0:\n",
        "            synchronize_q_networks(target_q_network, local_q_network)\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "flTWLf73c5i9",
        "colab_type": "text"
      },
      "source": [
        "## Solving the `LunarLander-v2` environment\n",
        "\n",
        "In the rest of this blog post I will use the DQN algorithm to train an agent to solve the [LunarLander-v2](https://gym.openai.com/envs/LunarLander-v2/) environment from [OpenAI](https://openai.com/).\n",
        "\n",
        "In this environment the landing pad is always at coordinates (0,0). The reward for moving the lander from the top of the screen to landing pad and arriving at zero speed is typically between 100 and 140 points. Firing the main engine is -0.3 points each frame (so the lander is incentivized to fire the engine as few times possible). If the lander moves away from landing pad it loses reward (so the lander is incentived to land in the designated landing area). The lander is also incentived to land \"gracefully\" (and not crash in the landing area!).\n",
        "\n",
        "A training episode finishes if the lander crashes (-100 points) or comes to rest (+100 points). Each leg with ground contact receives and additional +10 points. The task is considered \"solved\" if the lander is able to achieve 200 points (I will actually be more stringent and define \"solved\" as achieving over 200 points on average in the most recent 100 training episodes).\n",
        "\n",
        "### Action Space \n",
        "\n",
        "There are four discrete actions available: \n",
        "\n",
        "0. Do nothing.\n",
        "1. Fire the left orientation engine.\n",
        "2. Fire main engine.\n",
        "3. Fire the right orientation engine."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1BmMaLINc5i_",
        "colab_type": "text"
      },
      "source": [
        "### Google Colab Preamble\n",
        "\n",
        "If you are playing around with this notebook on Google Colab, then you will need to run the following cell in order to install the required OpenAI dependencies into the environment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wvzm9ShTc5jB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "a5f2faa7-c62b-4836-faf6-4e1fd8d5ab2b"
      },
      "source": [
        "%%bash\n",
        "\n",
        "# install required system dependencies\n",
        "apt-get install -y xvfb x11-utils\n",
        "\n",
        "# install required python dependencies (might need to install additional gym extras depending)\n",
        "pip install gym[box2d]==0.17.* pyvirtualdisplay==0.2.* PyOpenGL==3.1.* PyOpenGL-accelerate==3.1.*\n"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading package lists...\n",
            "Building dependency tree...\n",
            "Reading state information...\n",
            "The following package was automatically installed and is no longer required:\n",
            "  libnvidia-common-440\n",
            "Use 'apt autoremove' to remove it.\n",
            "The following additional packages will be installed:\n",
            "  libxxf86dga1\n",
            "Suggested packages:\n",
            "  mesa-utils\n",
            "The following NEW packages will be installed:\n",
            "  libxxf86dga1 x11-utils xvfb\n",
            "0 upgraded, 3 newly installed, 0 to remove and 35 not upgraded.\n",
            "Need to get 993 kB of archives.\n",
            "After this operation, 2,977 kB of additional disk space will be used.\n",
            "Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxxf86dga1 amd64 2:1.1.4-1 [13.7 kB]\n",
            "Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11-utils amd64 7.7+3build1 [196 kB]\n",
            "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 xvfb amd64 2:1.19.6-1ubuntu4.4 [784 kB]\n",
            "Fetched 993 kB in 1s (1,370 kB/s)\n",
            "Selecting previously unselected package libxxf86dga1:amd64.\r\n",
            "(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 144487 files and directories currently installed.)\r\n",
            "Preparing to unpack .../libxxf86dga1_2%3a1.1.4-1_amd64.deb ...\r\n",
            "Unpacking libxxf86dga1:amd64 (2:1.1.4-1) ...\r\n",
            "Selecting previously unselected package x11-utils.\r\n",
            "Preparing to unpack .../x11-utils_7.7+3build1_amd64.deb ...\r\n",
            "Unpacking x11-utils (7.7+3build1) ...\r\n",
            "Selecting previously unselected package xvfb.\r\n",
            "Preparing to unpack .../xvfb_2%3a1.19.6-1ubuntu4.4_amd64.deb ...\r\n",
            "Unpacking xvfb (2:1.19.6-1ubuntu4.4) ...\r\n",
            "Setting up xvfb (2:1.19.6-1ubuntu4.4) ...\r\n",
            "Setting up libxxf86dga1:amd64 (2:1.1.4-1) ...\r\n",
            "Setting up x11-utils (7.7+3build1) ...\r\n",
            "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\n",
            "Processing triggers for libc-bin (2.27-3ubuntu1) ...\r\n",
            "/sbin/ldconfig.real: /usr/local/lib/python3.6/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link\r\n",
            "\r\n",
            "Requirement already satisfied: gym[box2d]==0.17.* in /usr/local/lib/python3.6/dist-packages (0.17.2)\n",
            "Collecting pyvirtualdisplay==0.2.*\n",
            "  Downloading https://files.pythonhosted.org/packages/69/ec/8221a07850d69fa3c57c02e526edd23d18c7c05d58ed103e3b19172757c1/PyVirtualDisplay-0.2.5-py2.py3-none-any.whl\n",
            "Requirement already satisfied: PyOpenGL==3.1.* in /usr/local/lib/python3.6/dist-packages (3.1.5)\n",
            "Collecting PyOpenGL-accelerate==3.1.*\n",
            "  Downloading https://files.pythonhosted.org/packages/a2/3c/f42a62b7784c04b20f8b88d6c8ad04f4f20b0767b721102418aad94d8389/PyOpenGL-accelerate-3.1.5.tar.gz (538kB)\n",
            "Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from gym[box2d]==0.17.*) (1.5.0)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from gym[box2d]==0.17.*) (1.4.1)\n",
            "Requirement already satisfied: numpy>=1.10.4 in /usr/local/lib/python3.6/dist-packages (from gym[box2d]==0.17.*) (1.18.5)\n",
            "Requirement already satisfied: cloudpickle<1.4.0,>=1.2.0 in /usr/local/lib/python3.6/dist-packages (from gym[box2d]==0.17.*) (1.3.0)\n",
            "Collecting box2d-py~=2.3.5; extra == \"box2d\"\n",
            "  Downloading https://files.pythonhosted.org/packages/06/bd/6cdc3fd994b0649dcf5d9bad85bd9e26172308bbe9a421bfc6fdbf5081a6/box2d_py-2.3.8-cp36-cp36m-manylinux1_x86_64.whl (448kB)\n",
            "Collecting EasyProcess\n",
            "  Downloading https://files.pythonhosted.org/packages/48/3c/75573613641c90c6d094059ac28adb748560d99bd27ee6f80cce398f404e/EasyProcess-0.3-py2.py3-none-any.whl\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym[box2d]==0.17.*) (0.16.0)\n",
            "Building wheels for collected packages: PyOpenGL-accelerate\n",
            "  Building wheel for PyOpenGL-accelerate (setup.py): started\n",
            "  Building wheel for PyOpenGL-accelerate (setup.py): finished with status 'done'\n",
            "  Created wheel for PyOpenGL-accelerate: filename=PyOpenGL_accelerate-3.1.5-cp36-cp36m-linux_x86_64.whl size=1593669 sha256=5891e8777ee91ebe7e89bfa29026bc6bb9bd7a14d64b9efa05e400dc1e71c934\n",
            "  Stored in directory: /root/.cache/pip/wheels/bd/21/77/99670ceca25fddb3c2b60a7ae44644b8253d1006e8ec417bcc\n",
            "Successfully built PyOpenGL-accelerate\n",
            "Installing collected packages: EasyProcess, pyvirtualdisplay, PyOpenGL-accelerate, box2d-py\n",
            "Successfully installed EasyProcess-0.3 PyOpenGL-accelerate-3.1.5 box2d-py-2.3.8 pyvirtualdisplay-0.2.5\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NAlQ8Me7c5jH",
        "colab_type": "text"
      },
      "source": [
        "The code in the cell below creates a virtual display in the background that your Gym Envs can connect to for rendering. You can adjust the size of the virtual buffer as you like but you must set `visible=False`.\n",
        "\n",
        "**This code only needs to be run once per session to start the display.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HXeHSrZFc5jI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pyvirtualdisplay\n",
        "\n",
        "\n",
        "_display = pyvirtualdisplay.Display(visible=False,  # use False with Xvfb\n",
        "                                    size=(1400, 900))\n",
        "_ = _display.start()"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tynfeGv_c5jR",
        "colab_type": "text"
      },
      "source": [
        "### Binder Preamble\n",
        "\n",
        "If you are running this code on Binder, then there isn't really much to do as all the software is pre-installed. However you do still need to run the code in the cell below to creates a virtual display in the background that your Gym Envs can connect to for rendering. You can adjust the size of the virtual buffer as you like but you must set `visible=False`.\n",
        "\n",
        "*This code only needs to be run once per session to start the display.*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EntU0rXcc5jT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pyvirtualdisplay\n",
        "\n",
        "\n",
        "_display = pyvirtualdisplay.Display(visible=False,  # use False with Xvfb\n",
        "                                    size=(1400, 900))\n",
        "_ = _display.start()"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8NXANasAc5jd",
        "colab_type": "text"
      },
      "source": [
        "### Creating the Gym environment"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ssQ3ed61c5jg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import gym\n",
        "\n",
        "env = gym.make('LunarLander-v2')\n",
        "_ = env.seed(42)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jCbiT6-tc5jo",
        "colab_type": "text"
      },
      "source": [
        "### Defining a generic `Agent` and `train` loop\n",
        "\n",
        "In the cell below I define a fairly generic training loop for training and `Agent` to solve a task in a given `gym.Env` environment. In working through the hands-on portions of the [Udacity](https://www.udacity.com/) [*Deep Reinforcement Learning Nanodegree*](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) I found myself writing similar code over and over again to train the agent to solve a task. This is my first attempt to write something that I might be able to reuse on the course going forward."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2GqXNZCZc5jp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Agent:\n",
        "    \n",
        "    def choose_action(self, state: np.array) -> int:\n",
        "        \"\"\"Rule for choosing an action given the current state of the environment.\"\"\"\n",
        "        raise NotImplementedError\n",
        "        \n",
        "    def save(self, filepath) -> None:\n",
        "        \"\"\"Save any important agent state to a file.\"\"\"\n",
        "        raise NotImplementedError\n",
        "        \n",
        "    def step(self,\n",
        "             state: np.array,\n",
        "             action: int,\n",
        "             reward: float,\n",
        "             next_state: np.array,\n",
        "             done: bool) -> None:\n",
        "        \"\"\"Update agent's state after observing the effect of its action on the environment.\"\"\"\n",
        "        raise NotImplmentedError\n"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3vATxdcQc5jx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from numpy import random\n",
        "\n",
        "def _train_for_at_most(agentl: Agent,agentu: Agent,agentr: Agent,agentd: Agent, env: gym.Env, max_timesteps: int) -> int:\n",
        "    \"\"\"Train agent for a maximum number of timesteps.\"\"\"\n",
        "    state = env.reset()\n",
        "    score = 0\n",
        "    for t in range(max_timesteps):\n",
        "\n",
        "\n",
        "#        action = agent.choose_action(state)\n",
        "        au = agentu.choose_action(state)  #0\n",
        "        al = agentl.choose_action(state)  #1\n",
        "        ad = agentd.choose_action(state)  #2\n",
        "        ar = agentr.choose_action(state)  #3\n",
        "        #print(al,au,ar,ad)\n",
        "        if   (al==1 and ar==0): \n",
        "                    if ((au==0 and ad==0) or (au==1 and ad==1)):\n",
        "                        action=1\n",
        "                    elif (au==0 and ad==1):\n",
        "                        action = random.choice([1,2])\n",
        "                    elif (au==1 and ad==0):\n",
        "                        action = random.choice([1,0])      \n",
        " \n",
        "        elif (au==1 and ad==0):\n",
        "                    if ((al==0 and ar==0) or (al==1 and ar==1)):\n",
        "                        action=0        \n",
        "                    elif (al==0 and ar==1):\n",
        "                        action = random.choice([0,3])\n",
        "                    elif (al==1 and ar==0):\n",
        "                        action = random.choice([0,1])      \n",
        "#        au = agentn.choose_action(state)  #0\n",
        " #       al = agentl.choose_action(state)  #1\n",
        "  #      ad = agentm.choose_action(state)  #2\n",
        "   #     ar = agentr.choose_action(state)  #3\n",
        "\n",
        "        elif (al==0 and ar==1):\n",
        "                    if ((au==0 and ad==0) or (au==1 and ad==1)):\n",
        "                        action=3\n",
        "                    elif (au==0 and ad==1):\n",
        "                        action = random.choice([3,2])\n",
        "                    elif (au==1 and ad==0):\n",
        "                        action = random.choice([3,0])      \n",
        " \n",
        "        elif (au==0 and ad==1) :\n",
        "                    if ((al==0 and ar==0) or (al==1 and ar==1)):\n",
        "                        action=2\n",
        "                    elif (al==0 and ar==1):\n",
        "                        action = random.choice([2,3])\n",
        "                    elif (al==1 and ar==0):\n",
        "                        action = random.choice([2,1])      \n",
        "#        au = agentn.choose_action(state)  #0\n",
        " #       al = agentl.choose_action(state)  #1\n",
        "  #      ad = agentm.choose_action(state)  #2\n",
        "   #     ar = agentr.choose_action(state)  #3\n",
        "\n",
        "        else:\n",
        "                    action = random.choice([0, 1, 2, 3]) \n",
        "                    if   (action==0):\n",
        "                      al=0;ar=0;au=1;ad=0;\n",
        "                    elif (action==1):\n",
        "                      al=1;ar=0;au=0;ad=0;\n",
        "                    elif (action==2):\n",
        "                      al=0;ar=0;au=0;ad=1;    \n",
        "                    elif (action==3):\n",
        "                      al=0;ar=1;au=0;ad=0;\n",
        "\n",
        "\n",
        "        next_state, reward, done, _ = env.step(action)\n",
        "        agentl.step(state, al, reward, next_state, done)\n",
        "        agentu.step(state, au, reward, next_state, done)\n",
        "        agentr.step(state, ar, reward, next_state, done)\n",
        "        agentd.step(state, ad, reward, next_state, done)\n",
        " #       agent.step(state, action, reward, next_state, done)\n",
        "        state = next_state\n",
        "        score += reward\n",
        "        if done:\n",
        "            break\n",
        "    return score\n",
        "\n",
        "                \n",
        "def _train_until_done(agentl: Agent,agentu: Agent,agentr: Agent,agentd: Agent, env: gym.Env) -> float:\n",
        "    \"\"\"Train the agent until the current episode is complete.\"\"\"\n",
        "    state = env.reset()\n",
        "    score = 0\n",
        "    done = False\n",
        "    while not done:\n",
        "        au = agentu.choose_action(state)  #0\n",
        "        al = agentl.choose_action(state)  #1\n",
        "        ad = agentd.choose_action(state)  #2\n",
        "        ar = agentr.choose_action(state)  #3\n",
        "        #print(al,au,ar,ad)\n",
        "        if   (al==1 and ar==0): \n",
        "                    if ((au==0 and ad==0) or (au==1 and ad==1)):\n",
        "                        action=1\n",
        "                    elif (au==0 and ad==1):\n",
        "                        action = random.choice([1,2])\n",
        "                    elif (au==1 and ad==0):\n",
        "                        action = random.choice([1,0])      \n",
        " \n",
        "        elif (au==1 and ad==0):\n",
        "                    if ((al==0 and ar==0) or (al==1 and ar==1)):\n",
        "                        action=0        \n",
        "                    elif (al==0 and ar==1):\n",
        "                        action = random.choice([0,3])\n",
        "                    elif (al==1 and ar==0):\n",
        "                        action = random.choice([0,1])      \n",
        "#        au = agentn.choose_action(state)  #0\n",
        " #       al = agentl.choose_action(state)  #1\n",
        "  #      ad = agentm.choose_action(state)  #2\n",
        "   #     ar = agentr.choose_action(state)  #3\n",
        "\n",
        "        elif (al==0 and ar==1):\n",
        "                    if ((au==0 and ad==0) or (au==1 and ad==1)):\n",
        "                        action=3\n",
        "                    elif (au==0 and ad==1):\n",
        "                        action = random.choice([3,2])\n",
        "                    elif (au==1 and ad==0):\n",
        "                        action = random.choice([3,0])      \n",
        " \n",
        "        elif (au==0 and ad==1) :\n",
        "                    if ((al==0 and ar==0) or (al==1 and ar==1)):\n",
        "                        action=2\n",
        "                    elif (al==0 and ar==1):\n",
        "                        action = random.choice([2,3])\n",
        "                    elif (al==1 and ar==0):\n",
        "                        action = random.choice([2,1])      \n",
        "#        au = agentn.choose_action(state)  #0\n",
        " #       al = agentl.choose_action(state)  #1\n",
        "  #      ad = agentm.choose_action(state)  #2\n",
        "   #     ar = agentr.choose_action(state)  #3\n",
        "\n",
        "        else:\n",
        "                    action = random.choice([0, 1, 2, 3]) \n",
        "                    if   (action==0):\n",
        "                      al=0;ar=0;au=1;ad=0;\n",
        "                    elif (action==1):\n",
        "                      al=1;ar=0;au=0;ad=0;\n",
        "                    elif (action==2):\n",
        "                      al=0;ar=0;au=0;ad=1;    \n",
        "                    elif (action==3):\n",
        "                      al=0;ar=1;au=0;ad=0;\n",
        "\n",
        "\n",
        "        next_state, reward, done, _ = env.step(action)\n",
        "        agentl.step(state, al, reward, next_state, done)\n",
        "        agentu.step(state, au, reward, next_state, done)\n",
        "        agentr.step(state, ar, reward, next_state, done)\n",
        "        agentd.step(state, ad, reward, next_state, done)\n",
        "        state = next_state\n",
        "        score += reward\n",
        "    return score\n",
        "\n",
        "\n",
        "def train(agentl: Agent,agentu: Agent,agentr: Agent,agentd: Agent,\n",
        "          \n",
        "          env: gym.Env,\n",
        "          checkpoint_filepath: str,\n",
        "          target_score: float,\n",
        "          number_episodes: int,\n",
        "          maximum_timesteps=None) -> typing.List[float]:\n",
        "    \"\"\"\n",
        "    Reinforcement learning training loop.\n",
        "    \n",
        "    Parameters:\n",
        "    -----------\n",
        "    agent (Agent): an agent to train.\n",
        "    env (gym.Env): an environment in which to train the agent.\n",
        "    checkpoint_filepath (str): filepath used to save the state of the trained agent.\n",
        "    number_episodes (int): maximum number of training episodes.\n",
        "    maximum_timsteps (int): maximum number of timesteps per episode.\n",
        "    \n",
        "    Returns:\n",
        "    --------\n",
        "    scores (list): collection of episode scores from training.\n",
        "    \n",
        "    \"\"\"\n",
        "    scores = []\n",
        "    most_recent_scores = collections.deque(maxlen=100)\n",
        "    for i in range(number_episodes):\n",
        "        if maximum_timesteps is None:\n",
        "            score = _train_until_done(agentl,agentu,agentr,agentd, env)\n",
        "        else:\n",
        "            score = _train_for_at_most(agentl,agentu,agentr,agentd, env, maximum_timesteps)         \n",
        "        scores.append(score)\n",
        "        most_recent_scores.append(score)\n",
        "        \n",
        "        average_score = sum(most_recent_scores) / len(most_recent_scores)\n",
        "        if average_score >= target_score:\n",
        "            print(f\"\\nEnvironment solved in {i:d} episodes!\\tAverage Score: {average_score:.2f}\")\n",
        "            #agent.save(checkpoint_filepath)\n",
        "            break\n",
        "        if (i + 1) % 100 == 0:\n",
        "            print(f\"\\rEpisode {i + 1}\\tAverage Score: {average_score:.2f}\")\n",
        "\n",
        "    return scores"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2CHHq9c7c5j3",
        "colab_type": "text"
      },
      "source": [
        "### Creating a `DeepQAgent`\n",
        "\n",
        "The code in the cell below encapsulates much of the logic of the DQN algorithm in a `DeepQAgent` class. Since the `LunarLander-v2` task is not well suited for convolutional neural networks, the agent uses a simple three layer dense neural network with ReLU activation functions to approximate the action-value function $Q$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CxtRgm9Bc5j5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from torch import optim\n",
        "from torch.nn import functional as F\n",
        "\n",
        "\n",
        "class DeepQAgent(Agent):\n",
        "\n",
        "    def __init__(self,\n",
        "                 state_size: int,\n",
        "                 action_size: int,\n",
        "                 number_hidden_units: int,\n",
        "                 optimizer_fn: typing.Callable[[typing.Iterable[torch.nn.Parameter]], optim.Optimizer],\n",
        "                 batch_size: int,\n",
        "                 buffer_size: int,\n",
        "                 epsilon_decay_schedule: typing.Callable[[int], float],\n",
        "                 alpha: float,\n",
        "                 gamma: float,\n",
        "                 update_frequency: int,\n",
        "                 seed: int = None) -> None:\n",
        "        \"\"\"\n",
        "        Initialize a DeepQAgent.\n",
        "        \n",
        "        Parameters:\n",
        "        -----------\n",
        "        state_size (int): the size of the state space.\n",
        "        action_size (int): the size of the action space.\n",
        "        number_hidden_units (int): number of units in the hidden layers.\n",
        "        optimizer_fn (callable): function that takes Q-network parameters and returns an optimizer.\n",
        "        batch_size (int): number of experience tuples in each mini-batch.\n",
        "        buffer_size (int): maximum number of experience tuples stored in the replay buffer.\n",
        "        epsilon_decay_schdule (callable): function that takes episode number and returns epsilon.\n",
        "        alpha (float): rate at which the target q-network parameters are updated.\n",
        "        gamma (float): Controls how much that agent discounts future rewards (0 < gamma <= 1).\n",
        "        update_frequency (int): frequency (measured in time steps) with which q-network parameters are updated.\n",
        "        seed (int): random seed\n",
        "        \n",
        "        \"\"\"\n",
        "        self._state_size = state_size\n",
        "        self._action_size = action_size\n",
        "        self._device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "        \n",
        "        # set seeds for reproducibility\n",
        "        self._random_state = np.random.RandomState() if seed is None else np.random.RandomState(seed)\n",
        "        if seed is not None:\n",
        "            torch.manual_seed(seed)\n",
        "        if torch.cuda.is_available():\n",
        "            torch.backends.cudnn.deterministic = True\n",
        "            torch.backends.cudnn.benchmark = False\n",
        "        \n",
        "        # initialize agent hyperparameters\n",
        "        self._experience_replay_buffer = ExperienceReplayBuffer(batch_size, buffer_size, seed)\n",
        "        self._epsilon_decay_schedule = epsilon_decay_schedule\n",
        "        self._alpha = alpha\n",
        "        self._gamma = gamma\n",
        "        \n",
        "        # initialize Q-Networks\n",
        "        self._update_frequency = update_frequency\n",
        "        self._local_q_network = self._initialize_q_network(number_hidden_units)\n",
        "        self._target_q_network = self._initialize_q_network(number_hidden_units)\n",
        "        self._synchronize_q_networks()\n",
        "        \n",
        "        # send the networks to the device\n",
        "        self._local_q_network.to(self._device)\n",
        "        self._target_q_network.to(self._device)\n",
        "        \n",
        "        # initialize the optimizer\n",
        "        self._optimizer = optimizer_fn(self._local_q_network.parameters())\n",
        "\n",
        "        # initialize some counters\n",
        "        self._number_episodes = 0\n",
        "        self._number_timesteps = 0\n",
        "        self._number_parameter_updates = 0\n",
        "        \n",
        "    def _initialize_q_network(self, number_hidden_units: int) -> nn.Module:\n",
        "        \"\"\"Create a neural network for approximating the action-value function.\"\"\"\n",
        "        q_network = nn.Sequential(\n",
        "            nn.Linear(in_features=self._state_size, out_features=number_hidden_units),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(in_features=number_hidden_units, out_features=number_hidden_units),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(in_features=number_hidden_units, out_features=self._action_size)\n",
        "        )\n",
        "        return q_network\n",
        "        \n",
        "    def _learn_from(self, experiences: typing.List[Experience]) -> None:\n",
        "        \"\"\"Heart of the Deep Q-learning algorithm.\"\"\"\n",
        "        states, actions, rewards, next_states, dones = (torch.Tensor(vs).to(self._device) for vs in zip(*experiences))\n",
        "        \n",
        "        # get max predicted Q values (for next states) from target model\n",
        "        next_target_q_values, _ = (self._target_q_network(next_states)\n",
        "                                       .detach()\n",
        "                                       .max(dim=1))\n",
        "        \n",
        "        # compute the new Q' values using the Q-learning formula\n",
        "        target_q_values = rewards + (self._gamma * next_target_q_values * (1 - dones))\n",
        "        \n",
        "        # get expected Q values from local model\n",
        "        _index = (actions.long()\n",
        "                         .unsqueeze(dim=1))\n",
        "        expected_q_values = (self._local_q_network(states)\n",
        "                                 .gather(dim=1, index=_index))\n",
        "        # compute the mean squared loss\n",
        "        loss = F.mse_loss(expected_q_values, target_q_values.unsqueeze(dim=1))\n",
        "        \n",
        "        # agent updates the parameters theta of Q using gradient descent\n",
        "        self._optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        self._optimizer.step()\n",
        "        \n",
        "        self._soft_update_target_q_network_parameters()\n",
        "                 \n",
        "    def _soft_update_target_q_network_parameters(self) -> None:\n",
        "        \"\"\"Soft-update of target q-network parameters with the local q-network parameters.\"\"\"\n",
        "        for target_param, local_param in zip(self._target_q_network.parameters(), self._local_q_network.parameters()):\n",
        "            target_param.data.copy_(self._alpha * local_param.data + (1 - self._alpha) * target_param.data)\n",
        "    \n",
        "    def _synchronize_q_networks(self) -> None:\n",
        "        \"\"\"Synchronize the target_q_network and the local_q_network.\"\"\"\n",
        "        _ = self._target_q_network.load_state_dict(self._local_q_network.state_dict())\n",
        "           \n",
        "    def _uniform_random_policy(self, state: torch.Tensor) -> int:\n",
        "        \"\"\"Choose an action uniformly at random.\"\"\"\n",
        "        return self._random_state.randint(self._action_size)\n",
        "        \n",
        "    def _greedy_policy(self, state: torch.Tensor) -> int:\n",
        "        \"\"\"Choose an action that maximizes the action_values given the current state.\"\"\"\n",
        "        # evaluate the network to compute the action values\n",
        "        self._local_q_network.eval()\n",
        "        with torch.no_grad():\n",
        "            action_values = self._local_q_network(state)\n",
        "        self._local_q_network.train()\n",
        "        \n",
        "        # choose the greedy action\n",
        "        action = (action_values.cpu()  # action_values might reside on the GPU!\n",
        "                               .argmax()\n",
        "                               .item())\n",
        "        return action\n",
        "    \n",
        "    def _epsilon_greedy_policy(self, state: torch.Tensor, epsilon: float) -> int:\n",
        "        \"\"\"With probability epsilon explore randomly; otherwise exploit knowledge optimally.\"\"\"\n",
        "        if self._random_state.random() < epsilon:\n",
        "            action = self._uniform_random_policy(state)\n",
        "        else:\n",
        "            action = self._greedy_policy(state)\n",
        "        return action\n",
        "\n",
        "    def choose_action(self, state: np.array) -> int:\n",
        "        \"\"\"\n",
        "        Return the action for given state as per current policy.\n",
        "        \n",
        "        Parameters:\n",
        "        -----------\n",
        "        state (np.array): current state of the environment.\n",
        "        \n",
        "        Return:\n",
        "        --------\n",
        "        action (int): an integer representing the chosen action.\n",
        "\n",
        "        \"\"\"\n",
        "        # need to reshape state array and convert to tensor\n",
        "        state_tensor = (torch.from_numpy(state)\n",
        "                             .unsqueeze(dim=0)\n",
        "                             .to(self._device))\n",
        "            \n",
        "        # choose uniform at random if agent has insufficient experience\n",
        "        if not self.has_sufficient_experience():\n",
        "            action = self._uniform_random_policy(state_tensor)\n",
        "        else:\n",
        "            epsilon = self._epsilon_decay_schedule(self._number_episodes)\n",
        "            action = self._epsilon_greedy_policy(state_tensor, epsilon)\n",
        "        return action\n",
        "    \n",
        "    def has_sufficient_experience(self) -> bool:\n",
        "        \"\"\"True if agent has enough experience to train on a batch of samples; False otherwise.\"\"\"\n",
        "        return len(self._experience_replay_buffer) >= self._experience_replay_buffer.batch_size\n",
        "    \n",
        "    def save(self, filepath: str) -> None:\n",
        "        \"\"\"\n",
        "        Saves the state of the DeepQAgent.\n",
        "        \n",
        "        Parameters:\n",
        "        -----------\n",
        "        filepath (str): filepath where the serialized state should be saved.\n",
        "        \n",
        "        Notes:\n",
        "        ------\n",
        "        The method uses `torch.save` to serialize the state of the q-network, \n",
        "        the optimizer, as well as the dictionary of agent hyperparameters.\n",
        "        \n",
        "        \"\"\"\n",
        "        checkpoint = {\n",
        "            \"q-network-state\": self._local_q_network.state_dict(),\n",
        "            \"optimizer-state\": self._optimizer.state_dict(),\n",
        "            \"agent-hyperparameters\": {\n",
        "                \"alpha\": self._alpha,\n",
        "                \"batch_size\": self._experience_replay_buffer.batch_size,\n",
        "                \"buffer_size\": self._experience_replay_buffer.buffer_size,\n",
        "                \"gamma\": self._gamma,\n",
        "                \"update_frequency\": self._update_frequency\n",
        "            }\n",
        "        }\n",
        "        torch.save(checkpoint, filepath)\n",
        "        \n",
        "    def step(self, state: np.array, action: int, reward: float, next_state: np.array, done: bool) -> None:\n",
        "        \"\"\"\n",
        "        Updates the agent's state based on feedback received from the environment.\n",
        "        \n",
        "        Parameters:\n",
        "        -----------\n",
        "        state (np.array): the previous state of the environment.\n",
        "        action (int): the action taken by the agent in the previous state.\n",
        "        reward (float): the reward received from the environment.\n",
        "        next_state (np.array): the resulting state of the environment following the action.\n",
        "        done (bool): True is the training episode is finised; false otherwise.\n",
        "        \n",
        "        \"\"\"\n",
        "        # save experience in the experience replay buffer\n",
        "        experience = Experience(state, action, reward, next_state, done)\n",
        "        self._experience_replay_buffer.append(experience)\n",
        "            \n",
        "        if done:\n",
        "            self._number_episodes += 1\n",
        "        else:\n",
        "            self._number_timesteps += 1\n",
        "            \n",
        "            # every so often the agent should learn from experiences\n",
        "            if self._number_timesteps % self._update_frequency == 0 and self.has_sufficient_experience():\n",
        "                experiences = self._experience_replay_buffer.sample()\n",
        "                self._learn_from(experiences)\n"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r_00EcQkc5kE",
        "colab_type": "text"
      },
      "source": [
        "#### Epsilon decay schedule\n",
        "\n",
        "In the DQN algorithm the agent chooses its action using an $\\epsilon$-greedy policy. When using an $\\epsilon$-greedy policy, with probability $\\epsilon$, the agent explores the state space by choosing an action uniformly at random from the set of feasible actions; with probability $1-\\epsilon$, the agent exploits its current knowledge by choosing the optimal action given that current state. \n",
        "\n",
        "As the agent learns and acquires additional knowledge about it environment it makes sense to *decrease* exploration and *increase* exploitation by decreasing $\\epsilon$. In practice, it isn't a good idea to decrease $\\epsilon$ to zero; instead one typically decreases $\\epsilon$ over time according to some schedule until it reaches some minimum value.\n",
        "\n",
        "The Deepmind researchers used a simple linear decay schedule and set a minimum value of $\\epsilon=0.1$. In the cell below I code up a linear decay schedule as well as a power decay schedule that I have seen used in many other practical applications."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H8k3KG-vc5kF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def linear_decay_schedule(episode_number: int,\n",
        "                          slope: float,\n",
        "                          minimum_epsilon: float) -> float:\n",
        "    \"\"\"Simple linear decay schedule used in the Deepmind paper.\"\"\"\n",
        "    return max(1 - slope * episode_number, minimum_epsilon)\n",
        "\n",
        "def power_decay_schedule(episode_number: int,\n",
        "                         decay_factor: float,\n",
        "                         minimum_epsilon: float) -> float:\n",
        "    \"\"\"Power decay schedule found in other practical applications.\"\"\"\n",
        "    return max(decay_factor**episode_number, minimum_epsilon)\n",
        "\n",
        "_epsilon_decay_schedule_kwargs = {\n",
        "    \"decay_factor\": 0.995,\n",
        "    \"minimum_epsilon\": 1e-2,\n",
        "}\n",
        "epsilon_decay_schedule = lambda n: power_decay_schedule(n, **_epsilon_decay_schedule_kwargs)"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aY7N9N3uc5kN",
        "colab_type": "text"
      },
      "source": [
        "#### Choosing an optimizer\n",
        "\n",
        "As is the case in training any neural network, the choice of optimizer and the tuning of its hyper-parameters (in particular the learning rate) is important. Here I am going to more or less follow the Minh et al 2015 paper and use the [RMSProp](https://pytorch.org/docs/stable/optim.html#torch.optim.RMSprop) optimizer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lxAb-5wrc5kP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "_optimizer_kwargs = {\n",
        "    \"lr\": 5e-4,\n",
        "    \"alpha\": 0.99,\n",
        "    \"eps\": 1e-08,\n",
        "    \"weight_decay\": 0,\n",
        "    \"momentum\": 0,\n",
        "    \"centered\": False\n",
        "}\n",
        "optimizer_fn = lambda parameters: optim.RMSprop(parameters, **_optimizer_kwargs)"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ex8i7ipqc5kW",
        "colab_type": "text"
      },
      "source": [
        "At this point I am ready to create an instance of the `DeepQAgent`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kNFNnmAzc5kZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "_agent_kwargs = {\n",
        "    \"state_size\": env.observation_space.shape[0],\n",
        "    \"action_size\": 2, \n",
        "    \"number_hidden_units\": 64,\n",
        "    \"optimizer_fn\": optimizer_fn,\n",
        "    \"epsilon_decay_schedule\": epsilon_decay_schedule,\n",
        "    \"batch_size\": 64,\n",
        "    \"buffer_size\": 100000,\n",
        "    \"alpha\": 1e-3,\n",
        "    \"gamma\": 0.99,\n",
        "    \"update_frequency\": 4,\n",
        "    \"seed\": None,\n",
        "}\n",
        "\n",
        "\n",
        "deep_q_agentl = DeepQAgent(**_agent_kwargs)\n",
        "deep_q_agentu = DeepQAgent(**_agent_kwargs)\n",
        "deep_q_agentr = DeepQAgent(**_agent_kwargs)\n",
        "deep_q_agentd = DeepQAgent(**_agent_kwargs)"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K-OXulhCc5lJ",
        "colab_type": "text"
      },
      "source": [
        "### Training the `DeepQAgent`\n",
        "\n",
        "Now I am finally ready to train the `deep_q_agent`. The target score for the `LunarLander-v2` environment is 200 points on average for at least 100 consecutive episodes. If the `deep_q_agent` is able to \"solve\" the environment, then training will terminate early."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ciBNcVxrc5lL",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        },
        "outputId": "9bb4ff6e-7a40-4664-839f-b8b2571582f4"
      },
      "source": [
        "scores = train(deep_q_agentl,deep_q_agentu,deep_q_agentr,deep_q_agentd, env, \"checkpoint.pth\", number_episodes=2000, target_score=200)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Episode 100\tAverage Score: -155.05\n",
            "Episode 200\tAverage Score: -94.46\n",
            "Episode 300\tAverage Score: -62.65\n",
            "Episode 400\tAverage Score: -13.45\n",
            "Episode 500\tAverage Score: 74.15\n",
            "Episode 600\tAverage Score: 176.39\n",
            "\n",
            "Environment solved in 670 episodes!\tAverage Score: 201.19\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "41xWc_ZtvEVl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "86a80e85-b94d-41c0-a99f-b3d1631f2159"
      },
      "source": [
        "import numpy as np\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "np.save('/content/drive/My Drive/db/dqn/LL/scores_cs_1st',scores)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YEJQsx-Qc5lp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7CelfcJ6c5ly",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "scores = pd.Series(scores, name=\"scores\")"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ta2h2tEhc5l6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        },
        "outputId": "16446ada-b987-4150-8af5-629767224ec6"
      },
      "source": [
        "scores.describe()"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "count    671.000000\n",
              "mean       9.571663\n",
              "std      152.253236\n",
              "min     -483.267038\n",
              "25%      -83.542813\n",
              "50%       -2.552036\n",
              "75%      147.522446\n",
              "max      317.854118\n",
              "Name: scores, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "95vX1gAIc5mB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "0c4c6bb1-c595-4f6e-c9c6-83680c006df8"
      },
      "source": [
        "fig, ax = plt.subplots(1, 1)\n",
        "_ = scores.plot(ax=ax, label=\"Scores\")\n",
        "_ = (scores.rolling(window=100)\n",
        "           .mean()\n",
        "           .rename(\"Rolling Average\")\n",
        "           .plot(ax=ax))\n",
        "ax.axhline(200, color='k', linestyle=\"dashed\", label=\"Target Score\")\n",
        "ax.legend()\n",
        "_ = ax.set_xlabel(\"Episode Number\")\n",
        "_ = ax.set_ylabel(\"Score\")\n"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEGCAYAAACtqQjWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd5hU1dnAf+dO2wYsHQSUJiCKFcUaC5qImtgSWzRqNGos+TTFiOVL7CaWmNg+0djFEktsiAqIitIRQZTee9td2D4z93x/3Dl37r1z75Td2d0B7+959tmZW8+09z1vPUJKiY+Pj4+PTz7Q2noAPj4+Pj67D75S8fHx8fHJG75S8fHx8fHJG75S8fHx8fHJG75S8fHx8fHJG8G2HkBL0qVLF9m3b9+2HoaPj4/PLsXs2bO3Sim7NuXc3Vqp9O3bl1mzZrX1MHx8fHx2KYQQq5p6ru/+8vHx8fHJG75S8fHx8fHJG75S8fHx8fHJG75S8fHx8fHJG75S8fHx8fHJG75S8fHx8fHJG75S8fHx8fHJG75S8fHx8WkBVmyt4culW9t6GK3Obl386OPj49NWHP/AZABW3ndq2w6klWkzS0UIUSSEmCGE+EYIsUAIcXtiez8hxHQhxFIhxGtCiHBieyTxfGlif9+2GruPj4+Pjztt6f5qAE6QUh4AHAicLIQ4HPgb8A8p5UCgArgscfxlQEVi+z8Sx/n4+Oxm/PXdBQy8eVxbD8OnibSZUpEG1YmnocSfBE4A3khsfx44I/H49MRzEvtHCiFEKw3Xx8enlXjuq5XEdH+Z812VNg3UCyECQoi5wGbgE2AZUCmljCUOWQv0SjzuBawBSOyvAjq37oh9fHx8fNLRpkpFShmXUh4I9AYOA4Y095pCiCuEELOEELO2bNnS7DH6+Pj4+GRPQaQUSykrgU+BI4ByIYTKSusNrEs8Xgf0AUjs7wBsc7nWGCnlcCnl8K5dm7QcgI+PTwEg5e7hAtN/YK68tsz+6iqEKE88LgZOAr7HUC4/Txx2MfBO4vG7ieck9k+Su8u3zsfHJ4XdJa4S1fW2HkKr0paWSk/gUyHEPGAm8ImU8n3gz8DvhRBLMWIm/04c/2+gc2L774Gb2mDMPj4+rUQsXrhKZfmWava5bTwrttZkPDa+myjHbGmz4kcp5TzgIJftyzHiK87t9cAvWmFoPj4+BUBjXKeYQFsPw5VXZ66hLhpn3PwNXHP8wLTHRgtYObYEBRFT8fHx8XESi2fnNvrruwu498PvW3g0dipqGgHoVBrOeGy2ryMdHy/YyPrKumZfpzXwlYqPj09Bkm1M5bmvVvLkZ8tbeDR2KmqjALQvCrnut4Z78xEbuuLF2Zz5+JfNvk5r4CsVHx+fgiTahBn+81+t5JJnZ7TAaOxsr2kAIO6RK/S/7ywwHzdXqajssU07Gpp1ndbCbyjp4+NTkDQlFvGXdxdkPqiZ1EfjzFldCdhdW79/fS6rt9Xyxm+P5MVpq8ztzXV/NUUpLd60E4BB3ds1695NwbdUfHx8CoqAZnRfykcsoiVYuS2Z8WW1pt6as45ZqypSjj/2/slIKamPxtNetz4apzGW+ppjWaQkR+O6zeV25/vfceMb8zKe1xL4SsXHx6egCCRa+hVq1lRNQ1I5NGY5xn9OXMKQ28azsz6ask9Kia5Lhtw2nhMf+ixlv5elUlUXpe9NH/DclysYctt4fvPCbHNfXWOc0kjbZM757i8fH5+CIqAJiGc3Q883fxu/kLUVdTxyfkq1g0lNQ8x8HIvrfL9hB1OWbLVtc/LslysBQ9i3cwT3f/3cTD5dZLSUWr29NuXcuIfiUtlgd33wPXFdMuH7TckxNsbpmEVmWkvgWyo+Pj4FRVBTlkpuSmXe2spm3/uJyct475v1aY+pbUwqlWhc5/RHv+TuccmU5qq6VGtEbXOzOpRC8eLzJe77VXGo9Zq/eWEWcV1S2xijJNw2loqvVHx8fAqKQKBp7q+zHv8qb2NYtHEnm3fWu+6rtri/onFJo0P5qXRjN6JxnRVba8z4x9NfZE6F/p9X55qP566pNC0h530BPvluE5t21FPTEKck3DaOKF+p+Pj4FBRBM1Cfm1LJZ6+wnzz8OYfdPdF1n9NScbK12jv1d/66Ko5/YDJjPl9OXJfc9UFq0ebmHfV85bG2/RmPfcnvXv0aKaWnJacslVLfUvHx8fEBTTTN/WVlbUUtG6qaX4Fe5WJ1VDekVyrpKt/XVRj7Pl+yher6mOsxZzz2JRc8PZ3K2kbX1zBu/kb+O3cddR7ZZNG4Tm1jnJKIb6n4+Pj4NDmmYuXov33KEfdOAuCduevYsrNphYM1jamCv7YhjiYgEtRcraklm6tTtilUunRNQ5wdLplgAOurDLfbgXd8Yr4GJze89g07PZSSUnq+peLj4+NDMqai3FlxXXLW418y4btN6U5z5Z256/ifV+fy6+dmNmksDZa6kTXba3nw40VUN8QojQQJBzTXuMYTk5d5Xk/VqtQ2xng8zXHZ8LtXvnbdrjon+5aKj4/PD5aKmkZu/e986qNxgpohlpSlUtMYY87qSq5+eU7O11VB7lXbMreod6MhZiiB+micY/7+KY9MWsrCjTsoDgUIBTXXYsV07ExYETUNcV6ZsbpJY8qEes1ttdyUr1R8fHz4fPEW7hnXup1+rdz5wXe8NG01Hy3YaLqIlMCOxryznbLFLZPs+w076HvTB8xfW+V5XkNUpyEWZ8ht481tsbgkqAlCAcGmHe4ZYl6oOEqti1st3xzUp2OL38MNX6n4+Pjwq2dmMObz1u30a0UFxItDAdoVGW6bHQkB3BxlolBWz5rttVTWGm3rx83fAMDEhZvYWR/lu/U7Us5riOm8NM1uUVQ3xAgEBEFNY9W21GJFK5qAXx/Vz3Yu2Kvy802n0jD9upQyrHeHFrtHOtpyOeE+QohPhRDfCSEWCCH+J7G9kxDiEyHEksT/jontQgjxLyHEUiHEPCHEwW01dh8fn/wgpeSVGatZl8iYKisKmhXn2xKpudFY8904Kj5zzN8/5eSHvwAwA91lkSC/fm4mp/zri5TzGmLxlCythRt3EtQ0wkHNrIAf0LUUIKXgcOge7fnfnw5l/PXHAElLJR+K0ovaxhgJY69NaEtLJQb8QUo5FDgcuEYIMRRjmeCJUsq9gYkklw0eBeyd+LsCeKL1h+zj88MjUyPE5jB9xXZGvzWfhRuNrrqRoGbGArZVGxZFYzz3+4eDqaJNFRpuTLislNXQrijIzJWpjSDBcH/pLrEJTUAoIMxA/vC9OgFw5Y8G2I5Tp4YCxnh2NrS826s+qpsuxLagLZcT3gBsSDzeKYT4HugFnA4clzjseWAyxrr1pwMvSOMbN00IUS6E6Jm4jiuLFi3iuOOOs20755xzuPrqq6mtreWUU05JOeeSSy7hkksuYevWrfz85z9P2f/b3/6Wc889lzVr1nDRRRel7P/DH/7AT3/6UxYtWsSVV16Zsv/WW2/lxBNPZO7cuVx//fUp+++55x6OPPJIvvrqK26++eaU/Q8//DAHHnggEyZM4K677krZ/+STTzJ48GDee+89HnzwwZT9L774In369OG1117jiSdS9fIbb7xBly5deO6553juuedS9o8bN46SkhIef/xxXn/99ZT9kydPBuCBBx7g/ffft+0rLi7mww8/BODOO+9k4kR7cVnnzp158803ARg9ejRTp0617e/duzcvvfQSANdffz1z58617R80aBBjxowB4IorrmDx4sW2/QceeCAPP/wwABdeeCFr16617T/iiCO49957ATj77LPZtm2bbf/IkSO57bbbABg1ahR1dfYagtNOO40//vGPACnfOyjs797G5dso/9HFwKm2715NQ4z566oY1L0dLzz1eN6/e1urG9iYSMHtesZodAnLvnyfjRP+ywv/DTP90XZU1UXZuGEH3X7xV6SUPPHEE4x99bVEH60gG5cbn1OPC+4DoGr6W4g1c2xxCxGMcBe3A1D55Ssce+z9LNm8k+01jdzxSRlbqjW6nmm85orPnqNh3UIAThsL5SVhqgPt6fJT47PdPmEMVZWr0BDUNMaIhAJ8Pn8wDPsVMV1n2/hHiG5fB8DOcJDj3urAgCH7Qvkp1DTE2PreA8R2GsWN5SVhKmsbifQaQsdjLwFgy9v3EK+zu+KK9jqA8qPOB2DT63+BWANWVVc84DA6jDjL+CzH3kRJcBvHPRuETv2B5n/3cqEgGkoKIfpirFc/HehuURQbge6Jx72ANZbT1ia22ZSKEOIKDEuGSCTSYmP28fkhoHz/lS79rLJBFegVh9xrJpwBdF2X5uxexUG+35AUsOp6izbuYGd9jBH9OqdcMxgQxJB0Lg2zLbHsr5Od9VGzxiRTktSOuihaWer2RI0mZZEgoYRlUNsYp7w4hOrWpS6tCjqrHZaK1aA4UvuWPcVmnqQRt/LJAWIdF5ZM52ltMZpWT5UspYYiimnghODHnBZawRLZi7+JSjrr2yDeKf0LayFEW6WdmQMQogz4DLhbSvmWEKJSSllu2V8hpewohHgfuE9KOSWxfSLwZynlLK9rDx8+XM6a5bnbx+cHjZSSb9ZWcWCfcvre9AEAB/YpJ65L3rvuaABembGa0W/N57xD+3Df2fu7XufbdVVMW76NL5du5d6z9qdHhyJzn7ruyvtOdT333g+/ty0FPPY3I3h00lK+WraNbu0iTPnzCQy69UNz/9TRJ9CzQzGDbvmQxrjOzFtO5NC7J9iuGQ5o9OlUzJCe7flgnqcjw+T3Jw3ioU8WZzzOysBuZfTsUMQXS7Zy6VF9GdC1jFv/+y3nH7Ynd5+xH5MWbubyF2YxpEc7xl//I7ZVN3DIXRNSFN1p+/fk/XkbGCDWMS58MxERJSY1vpN7sV52oR21VFNMb7GFfbVVxNGYp/dnP7GCkDAU7Bq9K1voQFeq2ENsJSAkMQIEr5gEexyY0+tSCCFmSymHN+XcNrVUhBAh4E3gZSnlW4nNm5RbSwjRE9ic2L4O6GM5vXdim4+PTxMYO2M1t7z9Lc9ckpQdc9fYO/2qiXS6uedpj0wxHz8yaQl3nzks6zE4q8KlTN5r884GNlbZU3Z31MXo2cGIZzTGMTO5bNfAsHayjSrkqlCMcUpKEw0b20WCZoC+tjGGpglKHYWHoaA9phImio7gwLJKijot44aah6klwoUNozktMJXh2mIO0pZQTCMbZCeqKOWO6EUs7vpjpmwMECBOf7GBtbILdSSVeJAYJ2mzKenWjwebqFCaS5spFSGEAP4NfC+lfMiy613gYuC+xP93LNuvFUK8CowAqtLFU3x8fNKzZJMRy1i5NX1aLBiCOhty9Xs0RO1ZULqUtsD49a/Zq8ar6qKc/PDn1DQas3S3jsBSGuNQLicrQU3krfFkokaTsqKgaZ11LTNc7ipQrl5KWG/gx9pMNF1ySGQlZwe+IBivpWxOIxpxEPD36DnMlEOYGRviec8RxZ2A7cQJsET2TtkfI8iH+ggODpenntxKtKWlchRwETBfCKEirjdjKJPXhRCXAauAcxL7xgGnAEuBWuDS1h2uj88PDyWXs/WS5+pNd6bWxnVDfYUT1epqLXjFOU/akzeclkpAE0gM5eSWAFUcDnj2zMoF68ssi4Q4ckAXnvjlwRw/pJs5DoCIrIUZTxGZeAdjwkZsSEewk/Zskp1Yu8dhDD32F/zuha8Yrx+a9p4XH7EXVx8/ECHg6Ps+TZuW/EPN/pqCt4U60uV4CVzTooPy8fkBYSqMdMckfqItFXltcKQrG+4vSVGWLVAqHZZKQBPE4npCqaSKlxIPpdK1XSSrppP3nDmMm9+eb9tWGgmAlIzarwdE62D+OPosmsoH4U8YsmMNjNMRexzERStPRgdEaTfiXYcydfk2btlnH4YO6c+7iZcaCWq2fmNWLj+mP93bGxZReUmIzWnG6/baW4uCyP7y8fFpfUyFkc68yNFSsaqfeBZuJqcA1aURDwkHAxilbOl5eoq9C0BIE0TjoOsgXASrVxZa384lKUrlpKHd+cTSxPK9a4+2ZaIpNHR45xr47l1AQmM1XUWQNfTjzfDPOOeCK6DPCObdNYmquij9IqX0TAxNS1gU00aPJBrXOW/MNLMQ1Eku1kdbWip+mxYfnx8o2bi2zEB9E2yVJZt3ZjzGaY3EdSOmEnEpXnRj8SZ7m/lgoshQeri/ijyVSmnKtjMP6mUK5wN6d2BY7w6mEjB8bDqd2MHhX10Bc18GPQr7nwu/fJNFl37L2Y2382TkUuh7FASC7FFeDBjWiHrvVZv/Hh2K6NOphFDAWxmkMz5uPHmw7fkP0v3l4+PTtmSjMISHj2zzjnq6tS9KPcGCaoeSDtUFWKEnguzphGs6gppASuM6Xu4vN4b0bG+m9yqkhIAQxJGmRA9qgjBRfhKdyJVr36S8aB36liD89F9wyMXmudomQ6Fa37Ze5UV8v2GHrdrfKfxV5b0bboZfWSTIPj3bURS0vy43K6218C0VH58fKLlZKklmr6rgsHsm8s7c1Iz+dNd6ZsqKlBb0TveXNN1fqaLpDycN8r54gmBCGelSmtlZAIf1MwoBvdZt79G+iOMHd7OPBWm+RwIgHmXwkqeZHrmGPzf8iw4dOrBm6JVol46zKRSwKDTL+2GzVBLvrFOpBNMolTKXsd9w0iD+c9WR5utWNFEn5wVfqfj4/EBRs9m0gXoX4bRwoxFXmL5ie9b3qm6Iccf733H+mGm27SnuLymRUroqlZH7dE/Z5kStxWKkJScHr9xeRSF3kdejQyRFMEsJfbQtBIkxKLYInvwR+3z3EHP0vflD0e2I335Fn3P+DnuOSLme5mLgdSwJm2NU76tTqYRdtMEvR+zJtNEj6VAS8ryP8zp+TMXHxydnpJR8tGBjk5fdzaawMWnNuB/0l3e+dYwp+fiI/skWKqoppbPdS2qg3hDEYcuM/boTBgIQ8VAIVkKmpWJvgaKEtZd7aUDXMpvLqDNVHDB7NBO061ha9Cvuqfgj1O/g66Me57Lon5gdOCBtkMNU2JY3pDQSSIwtuS0gMlsq4aBm61Jgu0/if0izn9eW2V++UvHx2UWZtHAzV744myc/a9qytElLJU1MJU1KcX00zvNTV3meG7cIz5pEJbkzI8wZU5GJ4kerpfK7kXuz8r5TzaB2OpRQjuv2lGI1c/dyL5WXhBHALwMTmBj+A7OLfkvvNe/zPX0BWBEcAFd9QUWfEzOOwfZ6LI+LE+4rq1JxWkdusSSRpjeA+gydlomfUuzj45Mzai2PDVW5rT6oyCqm4nKMEnKZFqiKWSwo1UjRep1v1lSyaYc9jTeeaChptSiUMslGUKpjdd2e/aXcYk7F1KdTMc9dehhIyZDv/slPQ88wRw7ihdh+jLz4Vs4fu4m+9Qsp6X4AY0s6EdCMVpGZcuHcrMCShAsurktPZeBmSaV72WqfM6utKdl6+cJXKj4+uyhqwSevjKZM5DKXdRNRK7amrvtuFWbWDsSq27HVejn9sS9TztcTmVvWlGIvAeyGmvnHpbS5s0xLxXGNg/p0ZEBRDbxyGXsvHs/Y2PFMGXwzj190GACa9glz5UAO0yKu53uRjNMnX6/6nKzWmtP95apU0t7H2NupNGzb3pZ9gn33l4/PLsr6hIVS18RFtJQ809MUKSrh5BZTcWvmaMUa61HuL7cFr6zoiUC9m1WSlVLR3N1fQRf319n7tuf+0hfhiSNh+Wcs3P9Gbo5dTtwiFtU11JWcPb28KE4okEHd2pnbShJNJmO6TLleLq/Rijq6c5ldqWR6n1sS31Lx8dlFUYtQqRUScyWdr16hZtVWEWUqIxe5NXnRFmTCSohmcH+5odZTcXP5ZOP+CllTiq3uLxWo14wg/DmBz/j9xs8ILdsAPQ+An9zDqpqBMGO2bYzqGs5ixUx0a1fE2MtH2NaJt1oqXtlfbgpeS3NPdR2npZKnnplNwlcqPj67KErgey1ElYlsen+ZM94shdTmnQ1M+H4zJw3tTkyXFIcC1EXjpqViXtci9YRIKhsj+0u6KrycLRXL8QFNAySnrv8XdxQZK5Y2BvaCs56C/Y2etVqiJYtVIKt7etWVpOPIgV1sz4tDqe4vp55066Dsdsdk/YzxQKUrK9pynSxfqfj47KI4V0jMFSWs0rlK1L5cAr+qh1Y0plNeEqKuKp6SShzVk2MuCgZMF57q/aW5OOad8Qc3gpaUYuvhQ6pn8Fp4DCM2L2S6PoSP4ody0UX30a9rcklHs7bE8n4o60iNRymtpgTCVZwoLqVnOndMd/ksXV62Os+rTqUtLRU/puLjs4sQjeu8PnONOctXM950MRE3Kmsb6XvTB7z7zXogvUtKXTqXia8S5lFdmr7+rY5mjTFLEN+aPqzWU3GzVKyK5uvbTnK9t9U9JRAEiHNz8GUuXP4n9hKbmLDn9ZzbeBvPxEelpBe7WW7qnk2xVJwELJlpyToW+zGxeG6fpVVxvnPNUfzxx0bXgbaMqfhKxcenlWmM6Xy2eEvmAx38e8oKbnxzHv+ZvQZICo5sdMrcNZXMXmVUwC/dnFicK5ES7Hb6Qx8vou9NH5hFi1YZlUlgKTkXjet0SSxa9cyXK2zHWIWnNeNJT6yn4maUWAV6wKMPiVVRdGpYy7TItVwR/IDvO41kVMO9zOxxnjlCZ43Ifr2M+MeFh+9pbjMD9SqmEsguUO/GHuXFFIU0/vSTIZ7RLHf3V5qYimXfAX3K6d2xpMnjyxe+UvHxaWXuGfc9Fz8zg28cS/dmoiKRbaViKEq4x3VJ35s+4PevzfU894zHvuTsJ4wFrlIWd3KRQM9+uRKA2sQKi1Z3T6aW9koAx+KSzqWRxFjtx1jdXxGbpUIiUO9iqVhThD1cYSENOrGDI7QFnPv9tUSI8qfoFby/951U0N6mQIMOH1u3dkWsvO9UThiSbAcTMJVK8y2VolCAhXeO4tT9e5rbnO+kq1JxuaW5zbHPVHptWKfSpkpFCPGMEGKzEOJby7ZOQohPhBBLEv87JrYLIcS/hBBLhRDzhBAHt93IfXyajrIUdtSnLoWbDiXglLtLyR+lXN76OrXBoxtRh4slnfhRFoWURgX9UfdN4o3Za9NeX82eG+N6Sqqr87qQ6v7yaltvs1RcDiiigWs3jGZO0VW8Er6bSLyGXzbezH/ix9GuyAgfW2t6ssnksjWUtJzTXEvAq/1NzCU+5jbKZEzFvteaqNBWtLWl8hxwsmPbTcBEKeXewMTEc4BRwN6JvyuAJ1ppjD4+eUUFY7MJPFtRAk3JHTOmkqOEczZxdDtdbapPtFHZXtPIkNvGs66yjnlrq9JeP2mp6ESCmq2Pl9r+/rz15nPrfiOm4i5Ire+XU6l0pYJnQ/cztGYGAB/HD+GFg15jvuwPwIUj9uJPPxnMVccOMM9xur/cMLO/Eofmr/2J+3VyVQbOq5gdBX6oKcVSys+FEH0dm08Hjks8fh6YDPw5sf2FxLLC04QQ5UKInlLKDfj47EKoH3yua15oplLRE9fJPqZiRdW3KNxcJUq4NUSNe81aVZH19YUQicW2jJlzKCBotNRn/m38Qp76IhljCQWT78O05dvZWFXvKrxtKcIqKwudG4OvcnlgHALJS3vcyr+W92Qz5dwQ6QoY446ENK45fqDtek73lxtm9leee2l5pXPn6v5y7jMVpR9TsdHdoig2AsrB2QtYYzlubWKbDSHEFUKIWUKIWVu25B4M9fHxoqoumnMcxA3lvrLOlH/+xFf88T/fpD1PCdK4tFsouWZ/qZYp5nhcTlfXdjZ8zIYXp63i69WGMA8FRUqW1UcLNtmeWy2VSQs3GzGfDDJcE3CqNo3PwjdwVfB9ZsnBjGq8jznlJ7GZjoCwudDc3GXZWCrOivp84ZVS7GappG8oaX+ukh787C8PElZJTu+OlHKMlHK4lHJ4165dW2hkPj9Ejr5vkmu/qlxRs1GrnJu1qiJjrEJlPCn3l4p1WwWIlJLx33q3w4/rMtVSSatUcq+B+WZNJTe8biQNhDQtpZ/V5p32Bphua6cIBC9edhhjf5O6VgkAE+/gsfC/6KNt4Y7oRZzXeCuLZR+bANY0QWkihmJ1nan4SjYxFTOlOM9a5foTB9G/aylHDOhs297TpcV9uns7LSgzbdlXKjY2CSF6AiT+b05sXwf0sRzXO7HNx6dV2Gm2GmneD9YsKJTGgldu1sDXqyvoe9MHfLV0K/+ZZRjopqWS0CbKYrE2aZy0cDNXvTSbRyctdb13XTRutkxRpHN/1Texr1inRIV3KCBSFp5yKplwMLUhpibgmL27cuQAS1W6lByufcfLobthykN8HB5J//qXeCY+CjX3dwrgd649ittOG2pznb137dH87exhWbkfA440q67tjGy2340c6HFGdgzdoz2T/nAcHYrtC289esHBXDBiT9u2XPRZyMz+ajsKsaL+XeBi4L7E/3cs268VQrwKjACq/HiKT2thdUvEdZmV6yTTtdZX1fO7//ua8w7tk3LMW3OM+dIFT08H4KSh3c1ZqDJClHKLWzKptlYbRYbrK+tc713bEDPThE1cLRXjf1MsFeslgwGNkMMSyWa1wxR5v2EefHgjr4aNtGiO/TNHjbiBsz9YzH8sFp71NE0IBnZrx0BLU0eAvl1K6dulNKvXIcyYivG8KBRg5X2nZnVuU+hUGub0A/Zg7PTV1kFkHJ/z+Q82UC+EeAUjKN9FCLEW+AuGMnldCHEZsAo4J3H4OOAUYClQC1za6gP2+cFidSdF4xKXyXXWKKVSkag3+dylENJpTUCqa0Ndp9GiVJQw8Qos1zbGU7O/0oy1qUpF3SMUECmWiXNkbu4vmzFYsRJePBO0AA9Gf84E/RA+PP5qSkm6iw7r24m1FbX065pUFvlYUdeZ/dUaOBtIpkspdu5Tn/sPtveXlPJ8j10jXY6VwDUtOyIfH3h5+ioGdi1jhGU5XJtS0XWKabpWUcpA/d9Rn6pAdjq2RePJflExM/srdWxm/YJFTlsFTE1jLCUYvGpb6rooioYmur/UPYKWmMphfTsxY+X2lAyniIuGNl16tdvh5XNAj8GvP+KRBxbZjlMz8yMGdOaGk47gWUvlfgOdST0AACAASURBVD4ytswuxXkP1We+pyKXmIo614+p+PgUELe8/S3njplm22YtGIw6Zu/10XhOM0OnpeFmlTi7+sZ0HTWEuCNAb1UqapvVLWIde11j3BaDgdRsLCtNtVRilgy3Qd2Npo0DuhlWhNP95uywCyDijfDlv+D/jobty+Hcl6BLahzDOTO3ith8WBfOhpKtQTaxHq+UYjVet76UrYWvVHx8ssAquK0z7araKENuG88jHoFxN9TpTuFuxaloYnFpBugrahqZvGizq1JR5wkMZbdia42tLUt9VLfFYDIxt4kp1GpMAU1w71nDGHPRIQzp0R5ITZs977A+/O9pQ83nYaJcsvZ/4ZPboP0ecNlH0O8Y1/s413axCuT8WCoqpbg1LRVHzMnFPahwjkr4loqPT5JFG3fy5GfL2noYrljjENbHW2uMwPh/07RI+feUFfS96QMzlVe5r1RhoRtOpRKN66aFMn7BRi55diYVNdHEvqQAeXWGEeDVhOAv7yzg+Acm2zoE10VTLZWWQCmOgBCUhIP8eN8enskNkaDGr4/uR5AY5wUmMTVyLcNqpsIpD8DlE6DXIZ73MdvVoyy01H3NIZD0f7Ua6pblJSGuPLY/vz6qn+exTqsm30WaTaEQs798fqD87NEpNMR0rvhR/5yrzVsaL0tFkU5MP/3FcgAqa6OUhIOmayJdYaEzptIY11P6QqU0hgQ27jBqQDQBsxJdiVdYYib10Xir9IVS75E108urLiQU0GD1dGZFfku5MMb6fsdfcdphv8l4H2e2ky37Kw9aJf/tWTKj7lUcCjB61D5pj/V0f/1QA/U+PlaU/765KbtNZdmWak586DPXfbaYikWYJyujvX/EyVUN7bGUdPEKZ4HiyQ9/QccSe02DWxC9PmH9CCHoUhZh2ZYarn15jmV/6ygVZc1ZP0evtijt5j0Ln9/OTtmOP0WvZJJ+EMeX78FpWdwnxd1jkbL5+AYlV35sPXLRX85D9+xUQrtIkD/9ZEhex5QLvlLxKTjaKsf+7TnrPLvP2lOKmxYFVYpJuZ/cLJW4LglownUcFbX2rsb1aZSSENAlUahXYwmM17eS+0u9toBFkTgnCvuKlYzQvqfs07HQ/zjOW/Az1mF0wch2pm1aEC6WSj6sXed6Kq1BLq1hnK+xOBxg/u0/aYFRZY+vVHwKjrYy3aNpUmYaHXUqCun4nw4laJPV6qn3a4zpFIezS1d21ptY2Vkfc+0JlmugvqkoK8zq8upSOY8HQ09QTjV9undm0NYJAMR6HU7wF8+xbsEX5rHZWlPOFFp7TCWfgfrWI5dx5yNulG98peJTcLSUeyauS7ZVN9CtfWp/JYBozPu+1jRia2wjm2aOSkY0Wtx74N4CpSEWZ/X22ma/B159xJSlEg5orjGZfKHmBUX1W2Di47BuDkct/5Q6LUyUIO221jE2fiLvxUfw7EXXE4zY04qznVicMqwn/5ywhHMTXQlsvb/yklKs/rd+nUouqcWFhK9UdjOqaqM89MkiRp+yD0WhZpR9tyFO98yc1RUM7t6O0kjzvq7vfrOO0W/NZ9atJ1HmuNa4+RtSlry1YrVOrMI4F1eSmr0ri8UtpvLIpKX8e4r3OJpLXSKmEgm2jFIJEOcUbToVtCNMlGHj/gB1m6FdT5Ye8Cd+Nn0f6ghz60l7sakhxNTPlxMKhVKuk61S7d2xhAV3JJdkyrel0hbZX7m47Voz1TlbfKWym/HgJ4t4YeoqBvdon9KYbldBWmTd9ppGznr8K0bt14MnLvROLc2G9ZX11Ed1quqiKUrlaksw2w1b9pdFwcRycCWpFOKkckkV6s9/tTLr6zWF+qhOXJdGP66GzMfnwnCxkDcid9jvF+wHV3wKPQ9g3eIt1E43FtHaHgszetRg/nzyENe29G4ZdtnQUsWPbVlRn5bC0yl+ncruhhJ+LblG9bTl25i5cnuLXd86+9+e6I+1cOPOZl+3LhGwrktkVi3fUs3aitqszrXO6uesrmDGCuP1qxl1OoNF7VMWivnc4v46cZ9uQNOFKSQ71KajLhpHlzKrY3MhQJx/hR8FYFL8QC5pvJHrGq9lyVnjoecBxvgs0vLyo420ca8133NdI8aNvMRUNHtDydYgt5hK4WkV31LZzUg2mmu5L9t5iRYm+erWKqVkfVVyjQ2r60MpguI8uPJUexD1/4QHjfThbF6H1VJ5eMISHp6whJX3nWoqwGyUeENMtwlLq6XSw2UdjVxpVxQylbDnGKJxYnGZtko7FzpQzU3BVzg/+CkAv2u8hnH6CGIJ0XJVuNg8Vi3W1bNDER1L3deuVzQ1Q83m/srDS0zGN5p/rezvmX3GWeGpFF+p7HaYDQUL8dvmwUvTV3Pbf781n1trPnbWG2m0JVlmRKWjLmpYKCmt37Mg3aJXmVDCoSEWp85inViVinNdjabQriiYVqkENEF9zAjUOzsHuzGkRzsWbtxJmCh7iK0E0DlALONw7XsO1RZSJupoRx1FIsp78cOZpg/lXf1IrKLOWpuirJJsapCaaqnYA/V5iKm0gfsrFwrQUPGVyu6GW2plS/HBvA08Pnkp7193dLNqAiZ8Z29oaJ2lVtUZSiXbNNt0JC2V1AaOQqR3YbllhtUnZv2Q/lxFY0y3NYq0ur/Ki9PP3LMhkzXXoThkNJTUpW0JXyd/P3t/Pvt6AceH5zBp8zZuCL7J3lqyDU2DDLFGdmW6PpR6wrwSO57ZcrDrtay3US63UBYmhJel0qE4lH6CkefvvWiDOpXkvbM4pgCVna9UdjPUT7E12pxc98ocdGnMuJuTaVZZZy/qs87+dyQslcaYjpQy7ev6atlWhvXqQLuiEFt2NvDxdxv55Yi9zP1O95eVsnDQXNnRDbdMqc07GnKqqWmI6bZCxHWWhbQ6lDTfUsnk0upQHDID9V7HFlPPSesf55yNT4Ee5ecJXfdI7AxW6j1YT2em6ftgNOLP/B0L5GCp/O3sYWyraeTv4xd5JkB8fdtJae9na9OSl+yvxHVbUavkEg8tRI+EH6jfzVBCrik/qJvfns9f312Q9fHqq9/UCnNFVa3dZWOtQVSWyvQV2xl+1wSe+ny56zW2VjdwwVPTuf5VY230K1+cxS1vf8ua7clAvLJQ3JRKppiG22vcvLM+p8B6Q9RhqeTZ/ZXO+gBoXxRk6vJtzF1T6Skk7w+NoePXj8PgUXww8A6W6T25ofG3PBg7hzf1HzFV3xeJRnEou/motfhR6V+vdi3nHronxw0yEha8lLWmibQ9vVqsS3ErCu+c4qK+Umk+QoiThRCLhBBLhRA3tfV4Cg2vFeGyYez01TyXQ0qrule6yu5scLYfsQqU6oakAthW08jd4753vYZqwLh0SzW/emYGc1YbLdutAqjWzP5KVSrtPYS6rks+XrDRVRE1xnSzHX02BktDLO4ZzynPQanMuvVE87FV2GWyVKyv0dmcMkCc07UpnBaYRtVhv4dzX2Rht5MZ2fggb+upbeezjXFZs7uUAk6Xeab2NbX4026pNOkSNrQ26P2VC4Xo/tqllIoQIgA8BowChgLnCyGGpj+raUxfvo3XZ61piUu3KCrI3ZqLCjldQ7WNMc4bM5Wpy7axZWdqMcTY6auZsmSr+bzK6f6ySOhsA7bJ5Ws12/K8ui65duwcTn90iqlMnIL93nHfM3tVBUcP7GLb/sl3m3hl5mqueHE2L0xdmXLPuJQpbprD75nI3R98Z9umhGlDTGfzznrcyKWw0xpkt8Yn3JTKeYlKc4D2Rd6K687gs/wz/DjL9J7UHGossJpOXGXr7rRaKkoBp0sSSC6ZnNXlU7Aq2fz0/lL/W9P9ZZDulkqZFGKgfpdSKsBhwFIp5XIpZSPwKnB6S9zo3DHTuPGNeS1x6Sbz7bqqtEu/grUFeOt925yWypQlW5m2fDvnPzWNQ++ewJ/fmMfUZdvM/Te/PZ8L/z3d83pWRZLJv7x5Rz2j35pnrj/idAHFdMn78zbwzdoqi6Vij508mXCpRRxC+TcvzGL1NsN9tmlHA2WRoK1oMq7LFDfNxh31PPXFCtsxKiOrIaazapt7XUw22VgK6zit8Qk399fAbmXmY6t1oYY9XCzkidA/OC/wKS/HRnJq4z0EIolz0kisplgqB/Qu54IRe/KPcw/MeHyTLRVbRX2TLmEfTxsG6tOhfheFWKeyqymVXoDVfFib2GYihLhCCDFLCDFry5Yt7E6c9sgUjr1/ctpj8pX9tWVnA9+uqzKfPzF5GX1v+sD1x+5UKs7A+2uz1nD+U/bledNhs1QyyJZb/vstr8xYw+RFmwGMSnELizbuMB+nC9Q776uwpgCXhAN0LE3O9nUpbTEVt/b3FbWN5nvWGNNZubWGbonuwVbsLeLTf3hW5WE91s1SsW6zZtCFZT1naFMYG76bUYGZTNQP4s7YhdQTyaqJYiSUneiwracS0LjnzGH06VTieby6d9PdX/mNqYgs3ot807NDEaXhAKNHZW5fX4A6ZffL/pJSjgHGAAwfPrztVqppI9QLzvUHZfWx//yJr5i1qgJIFgY+MdlYLreyNrUOwtluZIdDqeSKNVCfKVaxtdpwr6ngb8QxW7/qpWT7FWWh1DoaOQY1QUyXroLMugJjSThA++IQa7YbWVuxuOTasV8nxildldXmHUn3X0MszsYd9fQsL2azxS347CWH2hRFIDEeLzRNEA5qNMZ0m4UTCaZaD1arpigUYB+xilMD07i88kOKwo1UylKOq7+X9SRdf0pRpfsKZfv98qqYz3R8UztV57v40VxPpRWld1EoYOtnlo4C1Cm7nFJZB/SxPO+d2OaTQDbRUqmxBMSVQlHXE0LQrijEjvoYFS5KxRlTqaxNVSpemUmvJJa/tWIVKOkWvwJjvXaASQuNWpdQ0PuFK2XiDNSr98pVqdRblUqQjiXJehJnLKjGJSXZGkNpiOpG+rXDojh+SDc2WjoKhAJaiqIetV8Pzhnex3yvi0MBGmO63f3lYqkoRfNjbSaXLbidmyOLAFgcGMjL9Ufxpb6vTaGANTid+l4e0Kecb3JYt94r08v7+OZZKlby4QJui4r6bCjkmMquplRmAnsLIfphKJPzgAvadkiFRbKi3v3bFovrBDSRMvPa2eBuXdQ2ximNBGlXZHxVtlW7KJUU91fqMaURdx/86Lfmp2w7f8w0c6GhdKLl04WbWZmIUXyz1nDVpYtNqPfGKfyV/MrGUrGuvmhVBIBZ52J9a5WC1YRh0UXjui0u069LKWB3fzln95P/eBx7lBfbXVmhAFV1UZvQtlolxdRzTuAzjv7qEe4PhvhpYCo18Z7cHb2A/8aPon2XXiyrcY/vpLNU/vLToZz1+Feu57mRq6Wi5VOp5CP7q0Ar6lVMpdCW3YZdTKlIKWNCiGuBj4AA8IyUMvvCih8AZkzFZV9clwy85UN+c0w/bjnVnjRX7VH4t7M+xuOTl5oNHd0slb+NX8jbVx9lPrdaPYpcspusRYhu2V/Kerr0uZkp+zLVaoA9TgKpy/xasSmVSNAm2NV68GAoP6WsSsPJ16rcWKWRIA2xONG44bI6aWh3+nct5Y8/NirRbRldjpTbbu0jKVaIimkkj5UcveUV9gnOZ39tGf3FBiIiRnX9AH4W2UhUFvFcr7/y1AKjHqc0jcxOt4RurotWZYoPOVGKcYAlwSAXrEI2V4XmhpaFK7AtKcRh7VJKBUBKOQ4Y11r303WZttiq0FBy0U1mKIviqS9WpCiVdRV1LmdAdUOUxz5dZj7f5tJb6uvVlVTWNlKecA25uazU7DyTO8uJ29ENMd0z1uIM1LtR2xhnW3UypqGu5RbHsLm/QgHbjNVpqahjVWbUqm01/PE/35jbGmI6jTGdcEDj/y6yt/EPpbFU3KzOomCA7mzntoYnEaEqBoj17LVmMwRhid6L5+I/4VP9IH5zyq8YObgrESTfvzgH2Oz5Wp33V7ctDgVMRWz9/KaNHsnh9070vA6Q82+nXVGIly4bwX692ud0nsJ6N6ursqkkU4qbfam8knR/FdjA2AWVSmsT1XUi2q6z2JX6zavZ97It1VTVRTl4z47EPJbL3bKzgcuen+W6z5lWvXq7u8tk1soKThza3RiDy35lqURzXMrWLWC7raaRo+6b5Hp8NjPj2sY4h9w1Iat77XS4v6whgk2WeMmGqnozxqJe62szk4mKpeFgwv0lXRVfyJbRlUExVq7hr3X3cETRVOpjYXZqRUSIMXvwDVz4zTDqiKDE65VCmBHrm0/Zh2VbalixtSateymZRivM1+O07hCCHh2KOLRvR2aurOC9a4+mLhrnnCenph97Fhy9d5fMB3lglbGdy5qvVJzvRaFRiKPylUoGYnFJMxccbDa/fm5m1uuXqJmkkhkjLe3dvZZgdwacrajKdMU0S72Jle0ubjErSaWSXfX9f79ex+eLtxBxKbLb7hLXyZaySDClTkWRKVDfs7zIVv3vdPM9lsiQc9a7AJRE7JaKE6sydLq/bPJszgvw2f0c0biayfEDeKvTZby3qRMCeGDwQdR9843nuf27ljHmokM46R+fZ8wus1IWCbC12njsPOvpXx3Kwo07GNa7g+f1WhOrJdkpQ3v9rK5XoMrEr1PZhUknBP/67gL63vRBi49h0sLNZhuSTKjZtpubyctSySV9c8nmatftMkNtSTgg+PMb8zjjsS+Tx6URbNe/Npe3vl7Hkk2pi3M9MmmJ+XhA11Lbvky9uDqXhb3rVFzOtc7Qh/Uqt80MnQH/FVuMwlQVdLf+3kvCQRqicRrjOmGXDLV0sQBNCIg1wJSH4d3roGoNj3a+mUuif2ZNZCASDR3NNYbg3KaywXJp2W+1opxtgDqUhBjRv7PtvM55EOZNxfqeu6VY50o+4jItSQHqlOyVihCiWAjh3t96NyaduyaXPlmthRqtm6LwEiS59O7yOtZ66biH8npt1hqbUqrxsBisuC25+7GlVX5J2G5G1mdYK6Vzadi19xdkFrQDupbaZoZO5aRiSqozr3XWXBI2UoBVoD4dyf2SM7QpBN67Dh49FCb8BQadDDetZm77E4xjXboAW3HOZFXAPyulkiZt1UuYzbntJD678fiM124p8i1j1VuaayywpSm0bDQrWSkVIcRPgbnA+MTzA4UQ77bkwAoFp6XyzZpK+t70AYstM2hnc77WoqouSlVKM8bEf5cheVV/q9d4aN+OGe9pvUaRpaq6Phrn9VlrkC79sLzwyjizopbgVQzoWmprPeJcQyTF9++gU2nEU5llWm0wFNBsAVvnuizKEnTrl+gM1KfjhOhn/F/oH8yL/IaHw48j5r8OJZ3horfhgtegqL05C1dWUVkkmBJPKgkH2N/hllJKJZfvrFWA9U+kQF98RF/XYzuVhimLBHnv2qO558xhWd8jX+R75q6ZxZj5vW6+KERLJdtowV8x+m5NBpBSzk3Uiuz2OAXkB/M3AEaNhKIxrptLpaZj6eZqdCkZ1L1ds8akUmoPuP1jwL4cbjKmIlPOsc5Oo3FJOCiobogxN1HM5pz1Z6IkHKQ+asQ37v9oEbWNcTqXhj1mwanf/kVZrDtfH9XpUhZmayKO0rEkbCvOdLYLqc+oVEJNFhDO+h7ndVRsypk9BQn3l0opdstQ++4dHg09RoQYJ9XMNhLmgXoZInLLRnAki5Qn6mWUC659UTDFKvnmLz9OsYpUvCeblv3Wy/UqL+agPcvpWBrOavnlYb07tFGcJb8pwOo9LTBDxWRXjqlEpZRVjm0F+jbnF1XB/Kf/fMOXS7e6HlMfzW7Wd+JDn/Hjf3ze7DGtr3LvdAup2V+Kr9dU2oT96u2G//+qF2dz+3tGV12vAkUvyiJBHjn/ICDpCorGdQ+BlbrtkmdT60ycGJZK8ofj7I7rnJ3XZfgs0q2NPuaiQ2zWlxNDqXhctyRkfg/cfugl4QANUSP7K8VS+fYteP1XHKF9x0HaEqZFjmS53oO/Rn/FqY33IFyyD08/0Gh5p97r/l3LUha/cnOzKUulV8filH1OrFf78qYTePSCgzOe09aotz5fwlZlf+WycFZrUoA6JWulskAIcQEQEELsLYR4BMi+rHYXJqYbLov/zF7LL59276zrdNG0NKc/OoXNO9wVi/ryO+X6WY9/ZRP2Jz70Oe/MXccMS1ZZrpZKUBMp7eJLwsG8VEMrGmI6moAjB3SmS1k45ccd0IRNSGeyVNLVLgzs1i7tkrxBTXgKqwODqzhNm8qeYhNd5VZYMgFhUezFoQDVCXeZWci44G144XR449ewx0GMaHiM4Q3/x0Od/pcTGh/iufjJLJO93G7HYf068fbVR3Lq/j0B2K9Xh6wEaSig8cQvD2bs5YdnPLYQBVa25Cu+rt6DQrVUCjG2kq0UuQ64BWgAxmJUtN/VUoMqJGJx6RnYVTRkaak0BbcMqa3VjRx2j3vRmZelAqnBWWsXYnBPhU1HwEXIGs0Q3d6Ppn35G6I6kaDG2N8YQvC8MfY6CE0IQgGB+ogyfVYdMyzbWxwKUIF7inWgcQcXLrqO88IbmKbvw2T9AC4NfsQ+7RvpXL0YlL7aCLwMR/S+nH+RCKgTIyBjxNHYo2YBvPk3mP86BMIw/Ndw4l+I/XUKkH0V+kF7dqR/1zLWV9Zx7QkDmbs6u55co4b1zOq4Qu4v5YUw/+fJUtGU+6swtUohfjYZlUpiYawPpJTHYyiWHxSNcZ3aaPqAsluGUr7IJkPKii7dLRVITSl2zspztTACmkA49NDsVRVMW55dTU021MfilAvrioWplooRz4qbx6ejPEOVdVGadULKJo6m885ZoEFfsZGLgokCyvYH8X78Z7xXNYAhYjUjOlRwZOl6jlj7NBPC45ihD+Hs2dO5IqJTJcvoM2sLiAAMOQ3O/jeE7EsZZxOfU3QoDnHXGUZA3Foz2bezd3v5bClEgZUJM+aVL0sl8b8wVUrhVfpDFkpFShkXQuhCiA4ucZXdnljcvaW5lWzcX651I3GdRyYt5fJj+tEusSpfXJf8e8pyLjx8L0rCwYwzbydmm5YsLJXicND2a8l1WeBgINVSeeiTxa7HNlVASWmXD854jaYJ2w/LrUOylUwFcU5Fe4BYyqmB6RykLSHy/WK+6HU5Vy87nEaC/DzwOcU9h3DrFVfy6tPTmVKxlY84lGM6d+HIS4cz5flb6bnibc4OfM7q8mNYsLmBwWIts4f8gUN+di2UdHIdQ679spLnGVqlb+cSJv+p7dJ62xLh+N/s6xV4oL4Qa+qzdX9VA/OFEJ8A5tKDUsrftcioCohoXHcV7NbvmJel8t+v13H9a3OZcfNIyopS3+r3523gnxOXUFHbyB2n72eec8+4hVTWRrnx5CG2NFdNuFsgcV0maxTMlGKZolicAvlv4xfanudqcRWHAlnPlJrzo7RmXCnF2JUKBmrr6V9fSa3cylfsiYZua1HiRpey1AWyrCSViuTJ7u/wk6rXAdguy2gccR1fyfPYuWwVAC/HT+SEsm6AvQpelxK0ALP6XMrDi44GJL8fOpiH1hkK9/6B+3OIh0KBpiuVfK/9UajV5OkwDZW8ZX8Z//1AffZkq1TeSvz9YFALN3kF562KpiGqo+uSpVuqbenCz09dCcCaijpmrEi6hFSTShVUtgaXVT+puIsby8s79cl3G9l3jw706VRic3851znJVJU/MMfOsHt2Km2VlEbrLbpGN/Bo6GlGadMJCAmJ5Vi2R8oopZ61sisT9IOpkqV0FjtZLbvxUvxESqlnB6VpBLaEbcu4pPZZTgtWcrw2l72qNvNSbCQvxU9ikyxn6onnICYusZ2lssWshYepISVh2++27okVZxZXtqTrLpwNL/z6MNZXJhuLFqC8yki+s78ocEulED+jrJSKlPJ5IUQYGJTYtEhK2bzl/QqcTKvvWVt0fLZ4Mw98vIi5ayq56PC9uOSovgzoWmb2jdqys95mFdTH4pSEg6aSsP4AdtSpugPDHZZNgPCql+YQCWosumuUraLeaXm4rdpo5dKj+nLc4K5c/fIcNqRJW1b07VySl5nSK785PO1yw5oQxq961Vc8uuM6SgJ1vBQbyTdyAIP6D6Ru7Tecrn9CDcWUinquDNpb59weeh6AO6MX0n7G1zy3dy0PL+1OT7GNQ7VFLJJ9ODvwOTyyiNMAgjBdH8LOYZfSZ7/LWPis0WzTLTFBFSFat5trXXgsbVuWoZlcrgtbJc9rXjzhR4O62p4X4iw4E2ZyQd6uZ1CgOqUg61SyUipCiOOA54GVGO9zHyHExVLK5hddFChBTdCQZn+NxVJ56osV5uMXp63ixWmr6NYuYs4cncqprlEplURTOMssdme9oavVoljZzpAaYjqVtY2W3l+pWWnpGkeCISAP2rNjyj2X3D2KvW/50HGsxi+G98n6S+3VuuWBXxxgK5IbJpazn7aCj+KH0l1U0ECI0Q3vwr9ugIoVaMU9OKvyz8yRxvzmnA69mbC+Pw/VGQV5QWJ0pBqJoEv3Xhy59T/8T/BNOohabgu9BNPgOOA4hxdsuyyDo29ADjuHY8csY3W9xmN7H8zI/smU6YBIrVNRtT02S8XlM7PG3o8a6N6FN6AJ4rpssqWiPot8CZrCE1dZYLq/8uUCNP772V/Zk63760Hgx1LKRQBCiEHAK8Ahac/yQAjxC4wq/X2Aw6SUsyz7RgOXYaTz/E5K+VFi+8nAPzFqjZ+WUt7XlHtnSyQUsCkOJ25Lx1rZvLOBdokZqVPIONemsH4vlItKLfSUy3f5wDs+MR8blopxnx8N6srni7ektHRxouICTv9xKKCx8r5TefKzZdz74UIGdS/j4xuOBbLPGHO64hQBDQJ12zlbM+Yn94eeRBOSe0P/Th4UBzqdAEf9jqL9zmZOIvXWON+ePBojyBbKAegdDvJMfBTPxE+mK5UcoX3HzddcSY9Swdh3PuCzRZuIo1FLhOn6Piw78WcIoFOnClbX3A77ZQAAIABJREFUVBpjsygLzWXFzD4dS8x9CrclnZWg71gSSingVIQCCaXS3JhKk85OZZeMqTj+N/96he7+KrzPKFulElIKBUBKuVgIkT7hPz3fAmcBT1o3CiGGYiwRvC+wBzAhocAAHgNOAtYCM4UQ70opv2vGGNKSqWYjk1IBzGI3Z0Geeu7m/lKWitqUSwdhK7pMNsPs1s6Ylmfqi5WpI6vbDDjdKd2o4PLgONbJLmyrOwYooT01HKwt4WBtMfP1/gz9/jOKPnqVB8NGYuEsfRCvx46lPbWsl50Zri1mYdkI/n7R77MekyIpnAVb6Mi7+lGMLusGHYpZ3PFHfKSvdD1PveOaEGZFtdfr3bNTieNeMHNlBYNv/ZCrjxtobssmiB4KaNRHdbMhZa4kl5jN7by7ztiP/XqltlTZBXVK/lOKC7T4Md8JCfkkW6UySwjxNPBS4vkvAfdVnbJASvk9uP7ATgdelVI2ACuEEEsxeo4BLJVSLk+c92ri2BZTKl6zSUU2zRDVF7EhRakYrirVyNEqqNR1lQXQ1O/y38YvNK9fmqi9yKRUMqVPmkurWn6xbkJSoHN+4FMuCYxnkLYOgG3b3+N34WLzucli0Psdy5NL2hOhkftj51KLUbcRDmiMix3OgFCp8xYmEu8flpuSzKWgz60ti3Nm2CehVJzKpyGm22pmsrEi2heF2FkfS7skQDqSbelzkzQXHr6X6/YClFcZUWPOlwuwULO/zM+6AD+kbJXKb4FrAJVC/AXweAuMpxdgjdiuTWwDWOPYPsLtAkKIK4ArAPbcc88mD8RpqfQqL2ZdZZ3p2shGqShue2eB7Xl9NG5zVSnBHIvr5sJP6dZFyRZVM1KScMNlamGi8LqjluXs6ILAJO4OPQPALdFfs1F25A/F7zNAX8arseOYqQ/hazmQwWIN55/2E44+/Ejuuzl1hejicIDGOj3t7D7d2+OqVMyJbJoXoWJdItXd5byk6iXmtmyudTGxbITcS5eP4N2562mMN63tT94FTSFKrCzJW5uWxPekcLsUF95nlK1SCQL/lFI+BGaVfdqEfyHEBKCHy65bpJTv5DTKHJBSjgHGAAwfPrzJXwWnQFIZOyo0kKkgMh3OOIQmBINu+dAWdzBb2DtegVetSjrU2LMtpPQS1NksWPQTbQZ3Bp9lid6LixpvYiPGAk5LIsewuqYG6/x3udyD09sP8JRdxaEAVXXRDALC+81Iq1TS6ZTEfzdF4dymrECnpQLGssfOsaS7b78upfzPiXvzwEeLvA9KQ75n04UnrjKT/HzzplWAwnN/KQrxM8pWqUwETsQoggQoBj4GjvQ6QUp5YhPGsw7oY3neO7GNNNtbBKfgLklk+SgLQgXUS8KBFAXTqTTM9hrv9N1UpZIayE4eYz+2JBzMyUpSY4TmKULI9EOVXBz4mFuDLzFXDuCXjTdTR7L9iOGKSz3f2U7eSnFi3OmsCiOpzH1/OvdXOtRn7KYonKgmnK6WSk0yfzB5rZYTA2rykG290Rc3Hp9WWBbgJDgjLZdSXFhapZA/m2yVSpGU0lyyT0pZLYRofnOhVN4FxgohHsII1O8NzMD4bPdOrOGyDiOYf0EL3N/E6XYqMzO5jO1bqxsoDQfoWV7MUscSuwO6lqZVKhWOLCxVJGlF3efbdTts24tCgZyVisokyxRTSWLcu0tZxAzyg4dLobGW3wdf51RtOgO0DXwaP4DrotfZFAp4r6CZrs2Vqm53/oDCAc1UwukSGTq59PnKJo6rLukWL/daSdHtdVi/A25Kx4umCoy9Opfy3KWHclg/72p9Kyoe5DmOgpwHpyf/FfVmUKWgKFTLCbJvfV8jhDAXUxBCDAfq0hyfFiHEmUKItcARwAdCiI8ApJQLgNcxAvDjgWuklHEpZQy4FqM78vfA64ljWwznh6Zm+1Yh1rdLqVlPYi0cK/Uobhs5xGjpcc3YObbtbgI3rkuicZ3rX5tr214czj0zSFlZ2XZTVi/xH+cewLj/OcbcrmbbpmWx5BN4fAS/C/6XHZTy9+i5XBq9kWpShZVzBU2FU0gP3yu5+qQS2E5L5t3rjqJ/Ym16r0D9zacM4djBXVO25yJr3CwVL92Q2f2VuH8WA2iOPDxucLeclzDwHMeup1OSKcX5rlPJy9V+GGT77bse+I8QYn3ieU/g3KbeVEr5NvC2x767gbtdto8DUqO5LYRzBqyEn9Vz1bdLqekGs36FvZTKiP6dmGhZMTL9/d1rQNKt9+GFGk+2loq6q9N9pN4DATD+Zpj2GHQZzDkNtzFD7pP2ml5KJeUeLu1MnIJ8SI/2XHXsAG58Y56npXLGgb2YviK1W3I2AfOkpeIdk0m5rsux1rY4tvcuEwUizQtjFDmS34zigi1+LJCviCtpp71CiEOFED2klDOBIcBrQBTDiliR7txdHedXyHR/WQT9kO7tzAJHq0wp85gpZlqb3IpXWmlxE2ahyv3lXFM9Eyk1GokX2S+23FAoB10IV32RUaFA9paK9Z4R01LxHlu69Gc3BZJToN7VUnE/MVPBYjZJDopCkReFLLi8yPcaMIWe/VWIZJJyTwLKhj8CuBmjCLGCRIbV7op1Bvzmb49g3z3aA3brYf8+5abgs/V2culIDLivTe5BXEpXgVnssdzt6FFDPK+lWomo5W7/78L0jRDUrMwpCNvVrubiwEfcUf1XKOkCJ90JQe8kQGtHYK+YilNIW1uUqOVwXYW7lhyrm/wIasI1zpEUOunSlN1ff7rzMsVMkq7DtIdlfUxrsEvHVPI09kJ1fxWY4WQjk5QLSCmVD+FcYIyU8k0p5W3AwDTn7fJICSP6deKrm07gkL06mULD2oreaKiYFBYn7tMdSPbtcpKTpSKla8aJl/vLy+UGlpTihPvriAGd07rRXFNqty9n5OQzuT30PBVaR7jobc/1QAC6lIV559qjPPcrvFxskHy/3MSDmzvynjOHJfd7Lf2bg6xpbkzFdp5L4agXBSPMC2QYuZCMqeTpeqZFXMBSvMDI5EsJCCGCiUD5SBJFhVmeu0sjpaRHhyL2KC8GkkLM+t0KWBaIEkLwyPkHsWxLtefKh5lanlvRdelqchd7rEyorBE3lAJRdSrGLD7dTN34bwrK6i3w+sWAxvmNt1DT4zDe7bl/2vGPHrVPVgVoTj0bdImppLMqrBal1UIMCHelks2Y0mV/WU/v3j5piWVyb2WTnmzeo0CEeYEMIyfUdyXfTTULTaUUynfEjUyK4RXgMyHEVoxsry8AhBADgd16FUhd2r+YyZlx8usV1DRLZ1gj3XffPTowa2WF6zVzsVTiunu6bHHI/SMrCnorlXBQIxzQTEvFqgw9zyFKl8WvwrwlRpZX5Sq+O+h2pk7dm2Ei/dfmoXMO4KyDe7NpR+b2+U6F4bbuiNsPyFTytm3266RzX6V7+cpCdHe7GdtOGdaDf5x7YMp43LAq8ZbO/sonhVitnYl8pxQXaqC+kEkrHaSUdwshJmJke30sk++sBlzX0oNrS3SHr17pA6v1oGnJ2azVZeE1a83FUolLiXSJbXulFKdb1zwU0AgFBKr2MRTQvGfW21dwiXybY8Mz6PXFUggWQacBcM4LrIsfBlPnuJ9nIZduuc4ZfNYxFRe3hHP9EtfsrSzGZFoqroF+Y1skGDDXUgHvz7xzaZhJfzyOuWsqs75/ocjyAhlGk8h3oN7XKdmTzRr1KasnSSndFyLfjZDSPlNTj61ZWUFNS5rbFpnulQmUi1KRUnpYKu4WSTAgKGcn7UUtndnBXDkAmQiZhWSUUwLTWacVoWshAoskf5JjKQtt57HY6SySeyLQYeKd8MUD3ADUiAibjrmb7j+6HEJGIaP4dmPivUg/9lyWtU0XUzGzv1zOU4fpevKx01Jx+xiyETbqXc8lJOOlVLq2i9ChOJSj+6swxHmBDCMnzJhKvgP1BaZUBnYrY0NVfU4ypbXYreMizUFKaRdSiW+XNfvLFlPJwlIJ5eT+yk2plNZt5P3ILfQWWwFYqXfn1fjxfKEPI/TiqdwvZ4MqMH8t0Y4gAD8LTGWzLCeOBl9sh/3P46RvfsSShg5MGHYc3UPJyvhss2KDObh6UrK/tKQ7MeySWeccizWZwSqMNeEVaDc/ME/c1rnxGm+2202LdheS1Hnvo9UK5L+i3vhfaG1aHvvlwcxdXWnLsCwUfKXigS7tX0zNdH85lUqqAPVauS9X95d7oN79Ixvadw82RTpT3WUQU9c00Fts5qbQq9zEq1DZldcCP+Wzun50KApy7/lHce6rK9lYKzg3MJmhYhWNBOl5zv2w75ls/GYCEEuxuLJ1a+WyAqFTAau1RDQhkjEoV4tDxbis2+z70wnDdDPZdJaKKWQcCt/LOlXfhV0zUF8gA8mJ/AbqKVD3V/uiUMryz4WCr1Q8kEiPQH3ymKBFqViP9VpkKZdAvZTuwUGvOpXi9p0ZcPMMAH5zk7FG+8naDE4MfcPPL32AJ59by/KaGnpHimHvE1gmJrBVNvD32HnmNVYOM5bkzVRRn0nqKUGazc/aKYvVW6QJYSpht/dBnWbUqbgLknRditNi3i77873qVLomZpK5FD8WCoWi3HIhC0O0SdcrMJ1S0BSeQ65A0KXTnWI8tmY0WRdxslkqHgIk02qSVuI5phS73XK8fhh3aldDl4Hs1dnox1Xk0aTRihLiTkGZbVPEXCwV5zWVQhYi6S50K5x0W1DMebd0xY/pSBtT8XhNXsvKn3ZAT8BSp5LFW7hrWgiFgUh5kJ/r+dlf2eMrFQ9SYiqJJ/PXJTOprbUQVgHqpVTyEVPxWpHSS9ipex7Qx1i3XSUapPvNmZaK45rJ+FF6gpq328pJSvaXlnw/laUS01PT4JJxm6RidxqIrqtSZhOob0JMxUvkdCpNWCq7ovurUAaSA9mkjOeCW32aT3p8peJBSkzFzb9uK35MbrfGVK78UX/zcS4uED3H7C8vlHXUr4vR1beitjFlvF44hXQmyyMUsAelsyt+dHdZaQLCievFXCyVY/buwsVH7MXdZ+5nbkupeUkTqE9rqXlcD7yFlVdvKPV57ZJ1Km09gCagxux3KW47fKXigWGppLq/nCRnRu4xlbMP6W05Nvv76x69v0JBjWcuGZ71dZSgL0+sLaL6f6VTEOq+QYdWySSQ1fFBzTtry0mq+ytpqaj30e19CAY0bj99P7q3T2anpbq/vJVCLiLn0L4dOXV/5cYytjmH5OUeUUsmaC7fEy8KxUAolHHkglt6eXNQ36+hPdvn54I/APxAvQdeFfVO3L7EVveXVbClE7LnDO/N67PWms/jursw1YTIadak3F8dS0JAsv+XcyTWoan0yVzdX8GAgGhua4c4f/zWdGT1OJ7B96Au4Xx/3WMimcd02v49eezTZeYiX/+5KrnAqZdS8GrBr2JguXUpLgxpXijjyIXkyo/5Gft+vTrwzjVHmQ1lfTLjWyoeOIWEl1Bwm71bjw1mqVQiwQBHDuicvL9HTMWrp5UXSqmUF9tXQXS6B6zX9Op9lSlQr+4VyMFS8Wqvr2nJivh0qzvazs0q+yuzG+oPJw1m3l9/TIeEIs4GL/dXialU1P0zX6tQLIRCGUcu5LtOBYx4ZLqOFT522uSdEkLcL4RYKISYJ4R4WwhRbtk3WgixVAixSAjxE8v2kxPblgohbmrxQTosFc9UUtNScVckdkvF+3ZCwMn79TCfe8VUNJFbDr66f3lpegHpdkUvReqZAZU4PpegdIr7y5L4oN7HbIOkztvmMg7nmNoXZa9QwFvxlYTUejv5DSC3BrvSWJ3sikkGuwttpX4/AfaTUu4PLAZGAwghhmKsP78vcDLwuBAiIIQIYKzjMgoYCpyfOLbF0D2yv5wkLRV3RWKNS6QvxrPv9yp+FEK4ds/1Qgm7do7W+M6h2CwVl23ZEFJKxRIXseKWFecU/NZAfSBLS8XL+shFsIy9fETWx0KqovMaolcKeDoKRSAWyDCaxC489F2eNlEqUsqPE+30/7+9M4+Pqjr7+PfJJCQhQUIIFmQNgqyJUTYRqBFJqFSliDaofSW8FheW0vettbhVXKjUrS4FAV8pVn1ZKi6I+BZR/NQFNewEFIKKNEhVoEHCYkhy3j/uncmdyayZSe6dcL6fz3xy59w79/7u5M55zvOcc54D8BHg7s0eCyxVSv2glPoS2AMMNl97lFJfKKWqgKXmsY1G/dFfITrqvfpU6r7W8D0V71xVtbXKb+dvpJ6KO62MiNA9K42Z5mJe9c5heTskO7OedqirOANd3eXyNiq+l/CXaSDBxxBZhwqHa1TqbiF0+CvQsRf2yArrGoEI1FHvmcDpvm4Y/zvnVIjx5125n5VIGl6a2OKEjvr/xFimGKAjhpFxU26WAfzTp9xv01JEbsRc96VLly4NFhVoRr0v/jqvret6hNunIuJd0dUo/3F6V4JEtLSptUJ+59Z8v7rAW//8Xwxg3+HjAefVBLqNpAR3n4rxvr6nkgB4zzlxy3CJUIPyGCbB4qn4X4m4Hr42JFhlGOtWeNghunCOcVgt7jQ9wahr+MSR6GZGo9lzEVkrIqV+XmMtx9wJVAMvxuq6SqmFSqmBSqmB7do1PDdO/Rn1/o/zZyi6mbPXoa71HujYutUNfTyVAH0qRgs+uHZrC706gAWqN1veoi0tOZE+foZQhppV7D5nXer4wLp8y9wtS399KmF31PsaykaoVwKdM5Shj2TynFOqw3gyJm6CZUPQNA2N5qkopUYF2y8ixcBlwCWWdVr2A50th3UyywhS3igopQKO6LJSl8W0DhHh9WnDqa6t9fZUfIzB7HH9uX/VTqipH9YKOPorDE8lOTGB4+biKbUBDvbtywhrprn72ADVnvtW3Zf0DfWckZrIkROnfD7j3bnvsoTDIh395auqKVurvhp/dUlPzvBaVjpE7NCCY/pU7BbQAIJlQ9A0DXaN/voJcBtwhVLquGXXSmCCiCSLSDbQE/gEKAF6iki2iLTA6Mxf2ZgalfJuaQf6oQcqz+nUmvO6tAk6T6VderL3kGTL7ppa/5MfW6UkhhzVZM0xFmiOh6+RjMWP0N/qmFZ+mnNWQB1uA2I1wnWeSvDr1g0jta8q8fXiLu7Vjl+OqMum0KG1sSz11PweIc/lEJviGOMWCXW5QONPe3PBrj6VPwPJwFvmg/uRUupmpdQOEVkO7MQIi01VStUAiMg04O+AC1iklNrRmAJrfWbUhxr9FagxnWhJkOhrVBLEmjrfe/5JrfJfOWekJnHsh+p65VasKfYD9Uf43k+rMIbQqjpXJeg5AxmVM1vVX/vBn4cChlfn8oTTwvRUfPtUgg2MCOuMgfFV5Gv4fLMRpCUnsnfOT6O8atMSj9WyJxlqPIpvJthiVJRSAZtrSqnZwGw/5auB1Y2py0qt8v5RBXpIQzWIXAnC9JE9GN2vvd9KTyzb9ftU6p+vdWoS+ytOBL2mdZlbf8kYwbv/4bwuGTz28zy/x/kj0C3Pu+58Fr3/Jb3b+599nGQxdu4MAu7vJNFjVMzOfhGPwQl3YEKCCBvvGhWwH6kxuWF4Nru/OcrW8gq++f6HqEYfOaU+jMfGfqgRiprGRw+884MnLhvB6K9g/KawF/07tvbrqXhCN3h7KoGyFCe6EkJ2+nqFvwJ5KhYpMy7p6Uk4GYxQq991bZvGvWP7B/TqWlgu+uCVuey4d7TnO3bfu3VIsXsIcuh5KpjngLbpyV75wEJ+KEa0SWvBwusHeiZNRrV+ikNqc4fIiIhIhm5rGgcnDCl2HHWjl8IxKuE/vPWGvIr3OhvWUxkJJRvW4k5OshoV/1bFGp5p3zqMSpi6BadyOrZukK4kVwJXDehEapILV4KQZpmQ6Rv+EqnzWgINNvDFt2M+PTnw4x11+CvA/ybQsgGR4JTqMB6H5QYaeahpOrRR8UOtx1OpKwsUzoikRRSsTyVBvM8VKEtxOFyWexal+78H6iY/+uKuvK8f2jVguMqXnj9qxcppw/wONw6HJFcCj1x9rt99vjnUEizhr1DfgyeJoM+/ok1aC35TcA6PvrW7QXobgr9nJ1Ic08h2io4IqPUTZdA0LTr85Ye6NCV1ZYFanpG0iIL2qfjMU/G38qN7TksoW3PTj7vz3m0Xe87jD7dRye8V2Vye3E4ZES02ZiXY51w+I70iSdPixp/X2Kt9qwhVRocnph+Vp+KMCtEZKiKjVg8pth1tVPzgr7UTqJKoq8hCV3z+MgNb07xYK5Pa2vqVaXKA9en9Xccd0vqPod3867b0XTQVwZZT9h3plWDpUwmV+t6Nv1uJdZd9qO+r1qK/4ddo8EdjSly29j1G3V4ZpzM6/OUH5efBDDX5sSFYR3z55v765uhJDh+r8jo+0FLC/khyJbBn9qUBdXvmgDThSKlgnorvJFLrEOtw56lEUpH/4oKurP/iEJ98eTjsz4RDLGL6TqkPnaIjEkJN0NU0PtpT8YO/jvpAj6i7NdeQ/g/BOvrLu2X41aHjTHlxk9fxKUnulRDDu1iiKyGkh9WUw2+T/CSUdDOq748APAtjJQiW1PcN91QCXbFdq2SW3zQ0rPNGQl34pTl4Kt5/4wF/DUJN06I9FT/UhTDqygJVa9GEObxn7Idu3Ua6Pn0w3E5Dk3oqQcJft43uzQ3Dsjlw5CTgXk44TE/F/NuUI34CSYpFpeaUVrZTdESCO+w72My0rWl6tFHxg7/WZqDO4mgqj4QE8VzD2r8SCHf4q2vb0HNKQuEeUhxuf0UsaBGio/7MM1L42mNUIu+o9/f9xbxPJcR+T59QM4h/xWNrv8eZ6ay7NZ+umS1DH6xpFHT4yw/+Mp36tuj/MC7H+zMNCn95b4eqh1LMmfKZaS2YPa5/5Be04K70Ao0OawzCGTVmHSQR8cqPDVYWO+66rC+tU5PISm8R+uAAOOE+wDk6IiU7Ky06o66JCu2p+EGZ8wV9c3EB9Dwznbf++6KYXEesM+oldCjNOvor2tCEu3sjFkZl1uV9Kf938NQxELxPxY01d1O4FUOoFTUbhQBf25icDozJ6RDVqR0z6sohMjTxhTYqfvA3ga1TGyPL7H8VnBOz69TLghziR2zN6RUt7tnqsTAqxcOywzouPE/F+GtdTyUamj4LWPMhHvtUNPajjYof6iY/1v2oGiPLrNfoMoun4koQv5V9SpjzVMLBXb83ZfirRZCOejfuMKO1oz5cmosBcUpV7hSHSRNf6D4VPyQI9O1wBplpoePi0fzuAs2oD9RCb2dJHR/tD76/mb+rU5um69CMxFMRCT8pY7CjguX/aginU0V7Gt2qJoZoo+KHjJYtWD1jBJefW39RqUCEyuDrD++ElXXhBn+V7+QR2dw2urfn/aX929O/Y8NycAH8xwVdWTV9OMN7ZjX4HJESWZ9KAzwVP/+CC89uy5wrc+rv0ITEMX07mrhCG5UoiVXiQOs8lUQ/le/kH3cntUVdn0pGyxasmj4iimuLx1tpKiL1VHwXugpIkP+BiDBhcJeA+/938hDW3Zof3nVOM7RN0TQEu5YTvl9EtonIFhFZIyJnmeUiIk+KyB5z//mWz0wUkTLzNdEO3bHGmFEv9bb9VabRTLJ0CpEMKU7wSVvTWFx4dlZYa8n40hDPNN6I/ydOYwd2eSoPK6VylVJ5wCrg92b5pRjr0vcEbgSeBhCRTOAeYAgwGLhHRNo0ueogNGSeitV2WD0Vf2GieP+Bvzp1WFjhrIHd2pDbqTV3jOnTgPBL41f07hBlU8wZzUpvwStTLmz8CwWgGbRjNDZg13LC31veplFXG4wF/qqMwPpHIpIhIh2AfOAtpdRhABF5C/gJsKTpVPsnmmGX3qO/LJ6KH6MS755KXueMsI5r2SKRldOGN7KahjOq75n8fGAnflPYq9GvNaJnO87rYmfbqc6L1mjCxbYhxSIyG7geOAJcbBZ3BP5pOazcLAtU7u+8N2J4OXTpEjiW7gQCzahP8hP+ilebMiX/bHqcmd7gz3fMSGXSsG6xExQlyYkuHrrK/0JjscIp/2un6NDEF41mVERkLdDez647lVKvKaXuBO4UkduBaRjhrahRSi0EFgIMHDiwyQLfDbmQ+I7+CuKpxOtInNt+0jv0QUH4YObIkMc8fd0AnnnvC7KzGm68NPWJzydOYzeNZlSUUqPCPPRFYDWGUdkPdLbs62SW7ccIgVnL341aZCyIJqGkz4x6t93w11EfzKZkpScH3nka0Kt9q4DLFGsaTqiGzKlTpygvL+fkyZNNpEgTa1JSUujUqRNJSUkxO6ct4S8R6amUKjPfjgU+M7dXAtNEZClGp/wRpdQBEfk78AdL53whcHuTim4EEnxyf7l/wpH0qWz9faHf4zWaaAn1VJWXl9OqVSu6desWt5706YxSikOHDlFeXk52dniplsLBrj6VOSLSC6gFvgJuNstXA2OAPcBxYBKAUuqwiNwPlJjH3efutI9nRKwLIQVPfR9oT+uWsWthaDRWQtmJkydPaoMSx4gIbdu25bvvvovpee0a/TU+QLkCpgbYtwhY1Ji6mhrflSWD/TTjffSXJnyacImbqNEGJb5pjP+fnlEfI8KtCP7+6x97tn1n1Ac7hf7tnn7of7kmHtFGJUoi/eH3at+q7rPivfKjRmMljhwWW5k9ezb9+vUjNzeXvLw8Pv74Y7slndbo1PcxomEJJeu2dfhL40b/q8Nn/fr1rFq1ik2bNpGcnMzBgwepqqpq8Pmqq6tJTNTVYjToby9KoolJemcpFh3+0sQt976+g51ffx/6wAjoe9YZ3HN5v6DHHDhwgKysLJKTjWH1WVlG1u2SkhJmzJjBsWPHSE5O5u233yYpKYlbbrmFDRs2kJiYyGOPPcbFF1/M4sWLefnll6msrKSmpobVq1czffp0SktLOXXqFLNmzWLs2LHs2LGDSZMmUVVVRW1tLStWrKBnz54xvefmgDYqNuJlKEIYDe3FsYy6AAAYoElEQVSpaJqaeBgwUFhYyH333cc555zDqFGjKCoqYujQoRQVFbFs2TIGDRrE999/T2pqKk888QQiwvbt2/nss88oLCxk9+7dAGzatIlt27aRmZnJHXfcwciRI1m0aBEVFRUMHjyYUaNGMX/+fGbMmMF1111HVVUVNTU1Nt+9M9FGJVY04AdoXZgrVPhLmxSNXYTTngnlUTQW6enpbNy4kffee49169ZRVFTEnXfeSYcOHRg0aBAAZ5xhrDv0/vvvM336dAB69+5N165dPUaloKCAzMxMANasWcPKlSt55JFHAGPo9L59+xg6dCizZ8+mvLycK6+8UnspAdBGJUqiqewTBBIS/HfUz7q8L7Ne31l3HW1VNBq/uFwu8vPzyc/PJycnh7lz50Z8jrS0uuUPlFKsWLGCXr28k4b26dOHIUOG8MYbbzBmzBgWLFjAyJGh0widbujRXzZirBliZoL1MRrFw7JJsyzKpecDaDT12bVrF2VlZZ73W7ZsoU+fPhw4cICSEmOu9NGjR6murmbEiBG8+OKLAOzevZt9+/bVMxwAo0eP5qmnnvKsQrp582YAvvjiC7p3786vfvUrxo4dy7Zt2xr79uIS7anYgIgRrxYBVwCjAtAhI5U931Y2sTqNU1A2d2rEQZcKlZWVTJ8+nYqKChITE+nRowcLFy5k0qRJTJ8+nRMnTpCamsratWuZMmUKt9xyCzk5OSQmJrJ48WJPB7+Vu+++m1//+tfk5uZSW1tLdnY2q1atYvny5Tz//PMkJSXRvn177rjjDhvu2Ploo2IDLhGqlfJKIumvI/6FG4ZwwYNvN7E6jd04zSl1mBwvBgwYwIcfflivPCsri48++qhe+V/+8pd6ZcXFxRQXF3vep6amsmDBgnrHzZw5k5kzZ0Yn+DRAh79iRCStOpenH8XbmPieo33rlOiFaTQaTROijUqUdDAr/rPbhb/OeaJl1qPbwOg+E41G0xzQ4a8oubBHFksmX8Dg7MywP+Me8VVbax395ewwg0aj0YSD9lRiwNCz23o8jnB44Gf9yUpPJi3ZZZmnIpyVkQrAFeee1RgyNXFCPEw61GgCoT0VGxib15GxeR0BvIYUt2uVzK4HfkILl7b1Gh0S1cQnttZeIvIbEVEikmW+FxF5UkT2iMg2ETnfcuxEESkzXxPtUx1b3EOK3R5LcqJLVyYaR2D3kGZNfGKbURGRzhjLAu+zFF8K9DRfNwJPm8dmYqxhPwQYDNxjWVo4rqmzH9qQaLzRlXpoXC4XeXl59O/fn8svv5yKioqgxxcXF/PSSy8BkJ+fz4YNGwAYM2ZMyM9GwuOPP05KSgpHjhyJ2TnjBTs9lT8Bt+E9knYs8Fdl8BGQISIdgNHAW0qpw0qpfwNvAT9pcsWNQN3oL//7by08hx+dUX+Clqb54jhH1XGC6khNTWXLli2UlpaSmZnZoBQtAKtXryYjIyNmupYsWcKgQYN4+eWXY3K+eEpeaUufioiMBfYrpbb6hHo6Av+0vC83ywKV+zv3jRheDl26dImh6sYhwRP+8v/DnTayJ9NG6sR1Gofz5kz41/bYnrN9Dlw6J+zDhw4d6kmdsmXLFm6++WaOHz/O2WefzaJFi2jTJnBwo1u3bmzYsIHKykouvfRShg8fzocffkjHjh157bXXSE1NpaSkhBtuuIGEhAQKCgp48803KS0trXeuzz//nMrKSubNm8fs2bOZNGkS8+fP5/PPP+fhhx8GYPHixWzYsIE///nPvPDCCzz55JNUVVUxZMgQ5s2bh8vlIj09nZtuuom1a9cyd+5c3nnnHV5//XVOnDjBhRdeyIIFCxCRgLpqamqYOXMm7777Lj/88ANTp07lpptuivCfEDmN5qmIyFoRKfXzGgvcAfy+Ma6rlFqolBqolBrYrl27xrhETBHP6C+NRtNQampqePvtt7niiisAuP766/njH//Itm3byMnJ4d577w37XGVlZUydOpUdO3aQkZHBihUrAJg0aRILFixgy5YtuFyugJ9funQpEyZMYMSIEezatYtvvvmG8ePH88orr3iOWbZsGRMmTODTTz9l2bJlfPDBB57zuvOTHTt2jCFDhrB161aGDx/OtGnTKCkpobS0lBMnTrBq1aqgup599llat25NSUkJJSUlPPPMM3z55Zfhf6kNpNE8FaXUKH/lIpIDZANuL6UTsElEBgP7gc6WwzuZZfuBfJ/yd2Mu2gZChb80mrggAo8ilpw4cYK8vDz2799Pnz59KCgo4MiRI1RUVHDRRRcBMHHiRK6++uqwz5mdnU1eXh5gpIHZu3cvFRUVHD16lKFDhwJw7bXXeip1X5YsWcIrr7xCQkIC48eP529/+xvTpk2je/fufPTRR/Ts2ZPPPvuMYcOGMXfuXDZu3OhJ03/ixAnOPPNMwOgvGj9+vOe869at46GHHuL48eMcPnyYfv36MWLEiIC61qxZw7Zt2zx9SEeOHKGsrIzs7Oywv4uG0OThL6XUduBM93sR2QsMVEodFJGVwDQRWYrRKX9EKXVARP4O/MHSOV8I3N7E0hsF9+gv3Ser0USOu0/l+PHjjB49mrlz5zJxYnSDQ61JJl0uFydOnAj7s9u3b6esrIyCggIAqqqqyM7OZtq0aUyYMIHly5fTu3dvxo0bh4iglGLixIk8+OCD9c6VkpLi8TxOnjzJlClT2LBhA507d2bWrFmcPHkyqBalFE899RSjR48OW38scNqEiNXAF8Ae4BlgCoBS6jBwP1Bivu4zy+Ied59SjbYqGk2DadmyJU8++SSPPvooaWlptGnThvfeew+A559/3uO1NJSMjAxatWrFxx9/DBghLn8sWbKEWbNmsXfvXvbu3cvXX3/N119/zVdffcW4ceN47bXXWLJkCRMmTADgkksu4aWXXuLbb78F4PDhw3z11Vf1zus2IFlZWVRWVnq8j2C6Ro8ezdNPP82pU6cAI93/sWPHovoewsH2yY9KqW6WbQVMDXDcImBRE8lqMtzzHPXwUY3T6NjGyPDws7z4yPBw3nnnkZuby5IlS3juuec8HfXdu3f3m504Up599lkmT55MQkICF110Ea1bt653zNKlS1m9erVX2bhx41i6dCm/+93v6NOnDzt37mTw4MEA9O3blwceeIDCwkJqa2tJSkpi7ty5dO3a1escGRkZTJ48mf79+9O+fXtPuCyYrl/+8pfs3buX888/H6UU7dq149VXX436ewiFNOfKbODAgco9Dt2p3PLCRt4s/RfzrjufMTkd7JbTbOk28w0A9s75qc1KQvPK5nL+a9lWxuadxRMTzrNVS1V1LUku8Tsh99NPP6VPnz42qLKHyspK0tPTAZgzZw4HDhzgiSeesFlV9Lr8/R9FZKNSamBD9NjuqZzuuIcS19Q2X+PuBCYM6kzrlkl2ywgLcdBYwBaJTouQ28cbb7zBgw8+SHV1NV27dmXx4sV2SwKcp0sbFZvxZCxuxh6jE5gzPtduCWGj4mLNxdOPoqIiioqK7JZRD6fp0s0Qm3GZjVJtVDS+OMdf0WjCRxsVm3GHv2prbRai0Wg0MUAbFZvRQ4o1Gk1zQhsVm9FDijUaTXNCGxWbqRv9ZbMQjePQzYzgHDp0iLy8PPLy8mjfvj0dO3b0vK+qqorptSoqKpg3b17A/bNnz6Zfv37k5uaSl5fnmYx4OqJHf9mMe/SXDn9p3DhpSLGTadu2LVu2bAFg1qxZpKenc+utt4b8XHV1NYmJkVV9bqMyZcqUevvWr1/PqlWr2LRpE8nJyRw8eDBqo9YQjU4hPlU3I9wrPurwlybeyc/Pr1f285//nClTpnD8+HHGjBlTb39xcTHFxcUcPHiQq666ymvfu+++G7GGZ555hoULF1JVVUWPHj14/vnnadmyJcXFxaSkpLB582aGDRvG1KlTue666zh27Bhjx47l8ccfp7KyEoCHH36Y5cuX88MPPzBu3DjuvfdeZs6cyeeff05eXh4FBQWeFPYABw4cICsry5MzLCsry7OvpKSEGTNmcOzYMZKTk3n77bdJSkrilltuYcOGDSQmJvLYY49x8cUXs3jxYl5++WUqKyupqalh9erVTJ8+ndLSUk6dOsWsWbMYO3ZsxN9JU6ONis3Ujf7SRkWjiZYrr7ySyZMnA3DXXXfx7LPPMn36dADKy8v58MMPcblcXHbZZcyYMYNrrrmG+fPnez6/Zs0aysrK+OSTT1BKccUVV/CPf/yDOXPmUFpa6vGMrBQWFnLfffdxzjnnMGrUKIqKirjooouoqqqiqKiIZcuWMWjQIL7//ntSU1N54oknEBG2b9/OZ599RmFhIbt37wZg06ZNbNu2jczMTO644w5GjhzJokWLqKioYPDgwYwaNYq0tLQm+CYbjjYqNjOqz4/46/qvGNA1024pGk1UBPMsWrZsGXR/VlZWgzwTX0pLS7nrrruoqKigsrLSK0Pv1Vdf7cn6u379ek8erGuvvdYTNluzZg1r1qzhvPOM9DiVlZWUlZUFXfAvPT2djRs38t5777Fu3TqKioqYM2cOAwYMoEOHDp48XWeccQYA77//vsfQ9e7dm65du3qMSkFBAZmZmR4tK1eu5JFHHgGMpJL79u1zfGocbVRs5sfntOOLP4zx9K1oNJqGU1xczKuvvsq5557L4sWLvQxVOC18pRS33357vRUS9+7dG/RzLpeL/Px88vPzycnJ4bnnnmPAgAER67dqVEqxYsUKevXqFfF57ESP/nIA2qBoNLHh6NGjdOjQgVOnTnlWUPTHBRdc4FnR0Tdd/KJFizz9K/v37+fbb7+lVatWHD161O+5du3aRVlZmef9li1b6Nq1K7169eLAgQOUlJR4tFVXVzNixAiPtt27d7Nv3z6/hmP06NE89dRTnv7WzZs3R/JV2IY2KhqNw3CvBtrCpX+ekXL//fczZMgQhg0bRu/evQMe9/jjj/PYY4+Rm5vLnj17POniCwsLufbaaxk6dCg5OTlcddVVHD16lLZt2zJs2DD69+/Pb3/7W69zVVZWMnHiRPr27Utubi47d+5k1qxZtGjRgmXLljF9+nTOPfdcCgoKPItt1dbWkpOTQ1FREYsXL/ZaGMzN3XffzalTp8jNzaVfv37cfffdsf2yGgmd+l6jcRjVNbU8smY3t1x0tqMzK8dz6vvjx4+TmpqKiLB06VKWLFnCa6+9ZrcsW4h16ntbmkIiMktE9ovIFvM1xrLvdhHZIyK7RGS0pfwnZtkeEZlph26NpilIdCUw89LejjYo8c7GjRvJy8sjNzeXefPm8eijj9otqdlgZ0f9n5RSj1gLRKQvMAHoB5wFrBWRc8zdc4ECoBwoEZGVSqmdTSlYo9E0D0aMGMHWrVvtltEscdror7HAUqXUD8CXIrIHGGzu26OU+gJARJaax2qjotHYiFLK76qQmvigMbo/7OwJnCYi20RkkYi0Mcs6Av+0HFNulgUqr4eI3CgiG0Rkw3fffdcYujUaDZCSksKhQ4d0Nog4RSnFoUOHSElJiel5G81TEZG1QHs/u+4Engbux8iZdz/wKPCfsbiuUmohsBCMjvpYnFOj0dSnU6dOlJeXoxtv8UtKSgqdOnWK6TkbzagopUaFc5yIPAOsMt/uBzpbdncyywhSrtFobCApKYns7Gy7ZWgchl2jvzpY3o4DSs3tlcAEEUkWkWygJ/AJUAL0FJFsEWmB0Zm/sik1azQajSY0dnXUPyQieRjhr73ATQBKqR0ishyjA74amKqUqgEQkWnA3wEXsEgptcMO4RqNRqMJjJ78qNFoNBovopn82KyNioh8B3wVxSmygIMxktNUxKNmiE/dWnPTEY+641EzGLrTlFLtGvLhZm1UokVENjTUWttFPGqG+NStNTcd8ag7HjVD9Lp1xjqNRqPRxAxtVDQajUYTM7RRCc5CuwU0gHjUDPGpW2tuOuJRdzxqhih16z4VjUaj0cQM7aloNBqNJmZoo6LRaDSamKGNih+cvCCYmdX5WxEptZRlishbIlJm/m1jlouIPGnexzYROd8mzZ1FZJ2I7BSRHSIyw+m6RSRFRD4Rka2m5nvN8mwR+djUtsxMG4SZWmiZWf6xiHRras0W7S4R2Swiq+JI814R2W4u2rfBLHPs82HRnSEiL4nIZyLyqYgMdbJuEekldYsjbhGR70Xk1zHVrJTSL8sLIw3M50B3oAWwFehrty6Lvh8D5wOllrKHgJnm9kzgj+b2GOBNQIALgI9t0twBON/cbgXsBvo6Wbd57XRzOwn42NSyHJhgls8HbjG3pwDzze0JwDIbn5H/Bv4XWGW+jwfNe4EsnzLHPh8Wjc8BvzS3WwAZ8aDb1OMC/gV0jaVm227IqS9gKPB3y/vbgdvt1uWjsZuPUdkFdDC3OwC7zO0FwDX+jrNZ/2sYq3jGhW6gJbAJGIIxQzrR91nByEs31NxONI8TG7R2At4GRmJk/xanazav78+oOPr5AFoDX/p+Z07Xbbl+IfBBrDXr8Fd9wl4QzEH8SCl1wNz+F/Ajc9tx92KGWM7DaPk7WrcZRtoCfAu8heHBViilqv3o8mg29x8B2jatYgAeB24Das33bXG+ZjCSy64RkY0icqNZ5ujnA8gGvgP+YoYb/0dE0nC+bjcTgCXmdsw0a6PSzFBGc8KR48RFJB1YAfxaKfW9dZ8TdSulapRSeRit/8FAb5slBUVELgO+VUpttFtLAxiulDofuBSYKiI/tu504vOB4d2dDzytlDoPOIYROvLgUN2Y/WpXAH/z3RetZm1U6hNsoTCn8o2Ya9SYf781yx1zLyKShGFQXlRKvWwWO143gFKqAliHETrKEBH3khFWXR7N5v7WwKEmljoMuEJE9gJLMUJgT+BszQAopfabf78FXsEw4k5/PsqBcqXUx+b7lzCMjNN1g2G8NymlvjHfx0yzNir1iccFwVYCE83tiRh9Fu7y680RHBcARywubpMhIgI8C3yqlHrMssuxukWknYhkmNupGH1An2IYl6sCaHbfy1XAO2aLr8lQSt2ulOqklOqG8dy+o5S6DgdrBhCRNBFp5d7GiPWX4uDnA0Ap9S/gnyLSyyy6BGMtKEfrNrmGutAXxFKzXZ1ETn5hjHjYjRFDv9NuPT7algAHgFMYLaUbMOLgbwNlwFog0zxWgLnmfWwHBtqkeTiGO70N2GK+xjhZN5ALbDY1lwK/N8u7Y6xGugcjdJBslqeY7/eY+7vb/JzkUzf6y9GaTX1bzdcO92/Oyc+HRXsesMF8Tl4F2jhdN5CG4ZG2tpTFTLNO06LRaDSamKHDXxqNRqOJGdqoaDQajSZmaKOi0Wg0mpihjYpGo9FoYoY2KhqNRqOJGdqoaJotIlLjk5E1aMZpEblZRK6PwXX3ikhWBMe/687Ma74fKCLvRqvDPFexiPw5FufSaMIhMfQhGk3cckIZaVbCQik1vzHFhOBMEblUKfWmjRrqISIupVSN3To08YP2VDSnHaYn8ZAY63d8IiI9zPJZInKruf0rMdZ/2SYiS82yTBF51Sz7SERyzfK2IrJGjHVX/gdjwpj7Wr8wr7FFRBaIiCuArIeBO/1o9fI0RGSViOSb25Ui8rB53bUiMtj0er4QkSssp+lslpeJyD2htJnnfVREtmKkptFowkYbFU1zJtUn/FVk2XdEKZUD/Bkjs68vM4HzlFK5wM1m2b3AZrPsDuCvZvk9wPtKqX4Yeau6AIhIH6AIGGZ6TDXAdQG0rgeqROTiCO4vDSO1Sj/gKPAARjqZccB9luMGA+MxsgRcbYbXgmlLw1g341yl1PsR6NFodPhL06wJFv5aYvn7Jz/7twEvisirGOk3wEg3Mx5AKfWO6aGcgbFw2pVm+Rsi8m/z+EuAAUCJkf6MVOoS9fnjAeAu4Hdh3BtAFfB/5vZ24Ael1CkR2Y6x5o6bt5RShwBE5GXzPqqDaKvBSP6p0USMNiqa0xUVYNvNTzGMxeXAnSKS04BrCPCcUur2sAQZhuoBjBX23FTjHVFIsWyfUnV5lmqBH8zz1FqyEkP9+1MhtJ3U/SiahqLDX5rTlSLL3/XWHSKSAHRWSq3D8BpaA+nAe5ghIrNf46Ay1oX5B3CtWX4pRlJBMBL0XSUiZ5r7MkWkawhdD2AssuVmL5AnIgki0hkjlBUpBea1U4GfAR80UJtGExLtqWiaM6lirNzo5v+UUu5hxW1EZBtG6/4an8+5gBdEpDVGi/5JpVSFiMwCFpmfO05dqvB7gSUisgP4ENgHoJTaKSJ3YaxomICRWXoq8FUgwUqp1SLynaXoA4wla3dipN7fFNE3YPAJRjirE/CCUmoDQKTaNJpw0FmKNacdYixiNVApddBuLRpNc0OHvzQajUYTM7SnotFoNJqYoT0VjUaj0cQMbVQ0Go1GEzO0UdFoNBpNzNBGRaPRaDQxQxsVjUaj0cSM/wehQo1hHVIRAgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RKZwI4NJc5mL",
        "colab_type": "text"
      },
      "source": [
        "#### Kernel density plot of the scores\n",
        "\n",
        "Kernel density plot of scores is bimodal with one mode less than -100 and a second mode greater than 200. The negative mode corresponds to those training episodes where the agent crash landed and thus scored at most -100; the positive mode corresponds to those training episodes where the agent \"solved\" the task. The kernel density or scores typically exhibits negative skewness (i.e., a fat left tail): there are lots of ways in which landing the lander can go horribly wrong (resulting in the agent getting a very low score) and only relatively few paths to a gentle landing (and a high score)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hz_4EBljc5mM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "06df62d2-101a-47ba-b070-2548b3a7a3dc"
      },
      "source": [
        "fig, ax = plt.subplots(1,1)\n",
        "_ = scores.plot(kind=\"kde\", ax=ax)\n",
        "_ = ax.set_xlabel(\"Score\")"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEGCAYAAABCa2PoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXgU15no/++rfd8FSEIgAQIsgQ024D22QxJjJzHOTDzGd+LrJM44k7Ez2W5u7GRuxnGuc+MkE9/JXCcZJ7HjZDLBSzwx4x/jfd+wZRsMAoOExKIFbQhJaF/e3x9dwo3Q0hJdXd3S+3mefqg+VXXq7Qbxqs45dY6oKsYYY8zpivI6AGOMMTODJRRjjDFBYQnFGGNMUFhCMcYYExSWUIwxxgRFjNcBeCknJ0eLioq8DsMYYyLK22+/3aKquaPLZ3VCKSoqory83OswjDEmoojIwbHKrcnLGGNMUFhCMcYYExSWUIwxxgSFJRRjjDFBYQnFGGNMUFhCMcYYExSWUIwxxgSFJRRjItCuunZ+8cJ+Xqls8ToUY06Y1Q82GhOJHio/zLf+9B4jSxnd9KFFfPvKM7wNyhjsDsWYiFLVdJzv/MdOLlqSw5vfXs9nzlvAvS9V88SuBq9DM8YSijGR5Hv/WUFibDR3X7uKOWkJ3P7JMpbPS+X7j++hd2DI6/DMLGcJxZgIsauunZcrW7j5siXkpMQDEBMdxT98vJS6Yz08tr3O4wjNbGcJxZgI8dvXDpAUF82mdQtOKr9wSTZn5KVx3ysH0JGOFWM8YAnFmAjQ0z/E1p0NXHVWPumJsSftExFuOH8hexs7ea+23aMIjbGEYkxEeO79Jrr7h7jqrPwx91+xMo+46Cge214f4siM+YAlFGMiwNZdDeSkxHPuouwx96cnxnLpslwef6+e4WFr9jLesIRiTJgbHBrm5X3NfHh5LtFRMu5xG1bMo6mzj4r6jhBGZ8wHLKEYE+Z21LbT0TvIxSWnrLh6kg8tzUUEnt/bFKLIjDmZJRRjwtzLlc2IwEVLciY8LiclnjPnZ1hCMZ6xhGJMmHtpXzNnFqSTmRw36bGXLctl++FjHO3qD0FkxpzMEooxYay7f5Adte1cMMndyYiLS3JRhW3VrS5HZsypLKEYE8Z2HG5naFhZW5QZ0PErC9JJjI1mW81RlyMz5lSWUIwJY+8cagPg7AWBJZS4mCjOXphhCcV4whKKMWGs/MBRlsxJISNp8v6TEeuKsnn/SAft3QMuRmbMqSyhGBOmhoeVdw4dY83CwO5ORqwrzkIVyg/aXYoJLUsoxoSpmtYu2nsGAm7uGrF6QQZx0VG8ac1eJsQsoRgTpnbV+SZ6XDk/fUrnJcRGc+b8dN46YAnFhJYlFGPCVEV9B3ExUSyZkzLlc1cVZrCrvoOBoWEXIjNmbJZQjAlTu+raOWNeKrHRU/8xPaswg/7BYfYe6XQhMmPG5mpCEZENIrJXRKpE5NYx9seLyIPO/m0iUuS37zanfK+IXO6UFYrI8yKyW0QqROQrfsffLiJ1IrLdeV3p5mczxk2qyq66dsoKptbcNWJVYQYA2w8fC2ZYxkzItYQiItHAPcAVQClwnYiUjjrsRqBNVZcAdwN3OeeWApuAMmAD8HOnvkHgG6paCpwH3DyqzrtVdZXz2urWZzPGbbVtPXT0DrIif3oJZX5mItnJcZZQTEi5eYeyDqhS1WpV7Qc2AxtHHbMReMDZfgRYLyLilG9W1T5VrQGqgHWq2qCq7wCoaiewByhw8TMY44mRDvkVBWnTOl9EOKswgx2WUEwIuZlQCoDDfu9rOfU//xPHqOog0A5kB3Ku0zy2GtjmV3yLiLwnIveJyJhjLUXkJhEpF5Hy5ubmqX4mY0Kior6DmChh6dzUaddx1vwMqpqP09lrDzia0IjITnkRSQH+BHxVVUdWE/oFsBhYBTQA/zTWuap6r6quUdU1ubkTry9hjFd2N3SwZE4KCbHR065j1YIMVGGnrTNvQsTNhFIHFPq9n++UjXmMiMQA6UDrROeKSCy+ZPIHVX105ABVbVTVIVUdBn6Fr8nNmIi0r7HztO5OAM5ynl9515q9TIi4mVDeAkpEpFhE4vB1sm8ZdcwW4AZn+9PAc6qqTvkmZxRYMVACvOn0r/wG2KOqP/WvSETy/N5+CtgV9E9kTAh09Q1S29bD0rlTf/7EX0ZSHAuzk6iotzsUExoxblWsqoMicgvwJBAN3KeqFSJyB1CuqlvwJYffi0gVcBRf0sE57iFgN76RXTer6pCIXARcD+wUke3Opb7tjOj6kYisAhQ4AHzRrc9mjJsqm44DUHKadygAZflp7KqzNeZNaLiWUACc/+i3jir7rt92L3DNOOfeCdw5quwVQMY5/vrTjdeYcLCv0fcw4rKgJJR0tu48QnvPAOmJsaddnzETichOeWNmsn1HOomPiaIwK+m061rhPBi5u97uUoz7LKEYE2b2NR1nyZwUoqPGvBmfkrJ833Ms1o9iQsESijFhprKxMyjNXQA5KfHkpSeceFDSGDdZQjEmjLT3DNDQ3huUDvkRZfnp7LImLxMCllCMCSNVTb4O+dMdMuxvRUEa+5uP090/GLQ6jRmLJRRjwsi+Rt+Q4dN9qNFfWX46qrCnwe5SjLssoRgTRvY3HSchNoqCjMSg1TkywaQ9j2LcZgnFmDBS09JFUXYyUUEY4TViXloC2clxNtLLuM4SijFhpKa1i+Kc5KDWKSKUFaTbHYpxnSUUY8LE4NAwh1q7KQpyQgFYkZ/GvsZO+gaHgl63MSMsoRgTJuqO9TA4rEG/QwHfE/ODw8q+I8eDXrcxIyyhGBMmqlu6AFjkyh2KbwqWXdaPYlxkCcWYMFHT7EsobjR5FWYlkpoQw057Yt64yBKKMWHiQGsXqQkxZCfHBb1uEWFFfjoV9sS8cZElFGPCRE2Lb4SXbx254FtRkMaehg4GhoZdqd8YSyjGhImRhOKWFQXp9A8OU9VkHfPGHZZQjAkDvQND1B3rcT2hANaPYlxjCcWYMHDoaDequJpQirOTSY6LpsISinGJJRRjwkCNM2TYzYQSFSWU5afbHYpxjSUUY8LASEJxY8iwv7KCNHY3dDA0rK5ex8xOllCMCQMHWrrISYkjLSHW1eusLEind2CY6mbrmDfBZwnFmDBQ7fIIrxHWMW/cZAnFmDAwMm292xbnppAQG2UzDxtXWEIxxmPH+wZp7uyjONf9hBIdJZTmpbHL7lCMCyyhGOOxAyMjvEJwhwK+Zq+K+naGrWPeBJmrCUVENojIXhGpEpFbx9gfLyIPOvu3iUiR377bnPK9InK5U1YoIs+LyG4RqRCRr/gdnyUiT4tIpfNnppufzZhgGZllOBR3KOBLKF39Q9S0doXkemb2cC2hiEg0cA9wBVAKXCcipaMOuxFoU9UlwN3AXc65pcAmoAzYAPzcqW8Q+IaqlgLnATf71Xkr8KyqlgDPOu+NCXsjdyih6EMBv6nsrdnLBJmbdyjrgCpVrVbVfmAzsHHUMRuBB5ztR4D14psZbyOwWVX7VLUGqALWqWqDqr4DoKqdwB6gYIy6HgCudulzGRNUNS1d5KcnkBAbHZLrlcxNIS4mKuQJxVaLnPncTCgFwGG/97V88J//Kceo6iDQDmQHcq7TPLYa2OYUzVXVBmf7CDD3dD+AMaFQ09IVsuYugNjoKM6YlxqykV69A0N8+Y/vsuwfnuAvfv4qTZ29IbmuCb2I7JQXkRTgT8BXVfWUnwpVVWDMHkcRuUlEykWkvLm52eVIjZmYqlLdfDxkzV0jVhSks6u+Hd+PirvueuJ9/nNHPX959nz2NHTy9Qd3hOS6JvTcTCh1QKHf+/lO2ZjHiEgMkA60TnSuiMTiSyZ/UNVH/Y5pFJE855g8oGmsoFT1XlVdo6prcnNzp/nRjAmOtu4BOnoHQ/JQo78VBel09g5y6Gi3q9fZ19jJ714/yGfOW8A//dVZ3HrFcl6pauGVqhZXr2u84WZCeQsoEZFiEYnD18m+ZdQxW4AbnO1PA885dxdbgE3OKLBioAR40+lf+Q2wR1V/OkFdNwCPBf0TGRNkI3N4LQphkxf4pmAB2FHrbj/Kr16qJi46im98dBkAm9YVMic1nvtfPeDqdY03XEsoTp/ILcCT+DrPH1LVChG5Q0Sucg77DZAtIlXA13FGZqlqBfAQsBt4ArhZVYeAC4HrgQ+LyHbndaVT1w+Bj4pIJfAR570xYa0mxCO8Riyfl0pCbBTvHmpz7RrHuvvZsqOeq1cXkOksaxwfE83Vqwt4ubKZ9u4B165tvBHjZuWquhXYOqrsu37bvcA145x7J3DnqLJXgDHXR1XVVmD9aYZsTEgdaOkiOkoozEoK6XVjoqM4c34G7x465to1tu48Qt/gMH997oKTyq9cmce9L1Xz1O4jXLOmcJyzTSSKyE55Y2aKmpYuCjMTiY0O/Y/i6gUZ7K7vcG047xMVRyjKTqIsP+2k8rPmp5OfnsAzexpdua7xjiUUYzwUqlmGx7K6MIP+oWEq6oM/fLi9e4DXqlq4fMU8fF2fHxARLliSw7aaozb9ywxjCcUYj6gqB1q6KM5J8eT6qxf4Zifa7kKz17PvNzI4rGwomzfm/vMXZXOse4A9R2zW45nEEooxHmns6KNnYIjinND2n4yYm5ZAfnoC7x4OfkJ5cV8zOSnxnDU/Y8z95y/OBuD1/a1Bv7bxjiUUYzxS3eJbNdGrOxTw3aUEe6SXqvL6/lYuWJxNVNSYY2jIz0ikICOR7S4kM+MdSyjGeORAi++hwiKP7lDA1zFf29YT1OlQ9jd30dTZxwXOXch4Vhak2wSVM4wlFGM8UtNynLiYKPLTEz2LYfUCX5NUMPtRXt/vewr+gsU5Ex63cn46B1q7ae+x51FmCksoxnikpqWb4uzkcZuFQqEsP52YKAlq09Nr+1spyEikMGviRDnytH6F3aXMGJZQjPFITctxT5u7ABJioynLT6P8QHD6UYaHlderWzl/cfYpw4VHG0ko71lCmTEsoRjjgcGhYQ4d7abIo2dQ/K0rzmJ77TF6B07/Acc9Rzo41j0waf8JQGZyHIVZiex0eT4xEzqWUIzxQG1bDwNDymIPR3iNOLc4m/7B4aA0e40MAz4/gIQCUJqXxvv2LMqMYQnFGA94NcvwWNYWZSECb9YcPe26XtvfyqKcZPICHGiwdG4qB1q7bTXHGcISijEe2N/sewZlUa73dyjpSbEsn5fGtprTe8hwYGiYbU7/SaBK5qYyNKwnEqyJbJZQjPFATUsX6YmxZCbFeh0KAOcWZ/H2wTb6B4enXcfOuna6+ocmHS7sr2SOL6Huazw+7eua8GEJxRgPVDd3sSg3edKRUKFybnEWvQPD7DyNEVcj/SfnLcoK+JxFuclERwmVjZ3Tvq4JH5ZQjPFAdctxFoVBh/yItcW+JPBG9fSbvV7b38Lyealkp8QHfE58TDQLs5PYZwllRrCEYkyIdfUN0tjRFxYd8iNyUuJZPi+Vlyubp3V+78AQ5QfaptR/MmLpnFQqm6zJayawhGJMiJ0Y4RUGz6D4u2RZLuUH2jjeNzjlc9852Ebf4DAXLQm8/2REydwUDrZ2n1b/jQkPllCMCbFqJ6EUh9EdCsAlS3MZHNZpTSn/SlUL0VHCuuLA+09GFGUnMzSsHG7rnvK5JrwElFBE5FER+biIWAIy5jRVNx9HxPcfaThZszCLpLhoXtzXNOVzX93fyqrCDFITpj5qbWS2gAM2dDjiBZogfg78N6BSRH4oIstcjMmYGa26uYuCjEQSYqO9DuUkcTFRXLA4mxf2NqMa+NK87T0D7Kw9xoXT6D8BTiyBbM+iRL6AEoqqPqOqfw2cDRwAnhGR10TkcyISHgPpjYkQNR6uIz+ZS5bmUtvWM6X/3N+obmVY4cJp9J8AZCbFkpYQw4FWSyiRLuAmLBHJBj4LfAF4F/hnfAnmaVciM2YGUlWqm4+zOAyekB/LpcvmAPDMnsaAz3mtqoXE2OgTa9RPlYhQnJtidygzQKB9KP8BvAwkAZ9U1atU9UFV/TIQnj8ZxoSh5s4+uvqHwmrIsL/CrCRWFKTx/+08EtDxqsqz7zdx/uJs4mKm38VanJ10YgVLE7kC/RfwK1UtVdX/o6oNACISD6Cqa1yLzpgZZn+zM8IrTJu8AD6+Mp8dh49x+Ojk/8G/f6ST2rYePlo697SuWZSTTH17T1Cm0DfeCTSh/O8xyl4PZiDGzAbVLeEzKeR4Pr4yD4D/2tUw6bHP7PY1ja0/Y85pXbM4JxlVOBRAEjPha8KEIiLzROQcIFFEVovI2c7rUnzNXxMSkQ0isldEqkTk1jH2x4vIg87+bSJS5LfvNqd8r4hc7ld+n4g0iciuUXXdLiJ1IrLdeV056ac3JsQqG4+THBdNfnqC16GMa0G2r9nr8fcmTyhP72lkVWEGc1JP7/OMDKGubrZ+lEg22R3K5cBPgPnAT4F/cl5fB7490YkiEg3cA1wBlALXiUjpqMNuBNpUdQlwN3CXc24psAkoAzYAP3fqA/itUzaWu1V1lfPaOslnMybk9jV2smRuathMCjmev1g9n/dq29k1wWSR+5uP815tO1eunHfa1zvxLIqN9IpoEyYUVX1AVS8DPquql/m9rlLVRyepex1QparVqtoPbAY2jjpmI/CAs/0IsF58P2kbgc2q2qeqNUCVUx+q+hJw+isBGeOBfY3HWTonfJu7RvzlOfNJiI3iD9sOjnvMo+/UEiVw9aqC075eemIsGUmx1uQV4SZr8vqMs1kkIl8f/Zqk7gLgsN/7WqdszGNUdRBoB7IDPHcst4jIe06z2JhjGEXkJhEpF5Hy5ubpTYRnzHS0dfXTcryPpXNTvQ5lUumJsVx1Vj5/freetq7+U/b3DQ7xcHktH1qay5y04DTfLchKCmgggAlfkzV5jQxFSQFSx3iFk18Ai4FVQAO+prlTqOq9qrpGVdfk5uaGMj4zy41M0V4yN/zvUABuvGgRvYND3Pty9Sn7HtteT1NnH5+/sDho1yvMTKK2rSdo9ZnQi5lop6r+q/Pn96ZRdx1Q6Pd+vlM21jG1IhIDpAOtAZ47OtYTT2KJyK+Ax6cRszGu2edM0R4JdygAy+al8skz87n/1RquXVN4op+ju3+Qnz1bSWleGheXTO/p+LHMz0rkqd1HGBpWoqPCu4/JjC3QBxt/JCJpIhIrIs+KSLNfc9h43gJKRKRYROLwdbJvGXXMFuAGZ/vTwHPqm0RoC7DJGQVWDJQAb04SY57f208Bu8Y71hgvVDZ2khofQ14Yj/Aa7bYrlxMbHcVXNr9LZ+8AqsoPtu6htq2Hf/xkaVAHFyzISmJgSGns6A1anSa0An0O5WOq2gF8At9cXkuAb050gtMncgvwJLAHeEhVK0TkDhG5yjnsN0C2iFThGzl2q3NuBfAQsBt4ArhZVYcAROSP+J6BWSYitSJyo1PXj0Rkp4i8B1wGfC3Az2ZMSPhGeKWE/Qgvf3npifzkmrOoqO/gk//yCtf+6xv82xuH+JuLizl30fQmgxxPYabvSQTrmI9cEzZ5jXHcx4GHVbU9kB8KZ+ju1lFl3/Xb7gWuGefcO4E7xyi/bpzjr580IGM8VNl4nI+ccXpPlHvh8rJ53P+5tdz99D6OdvfznSvP4MaLgtd3MmJBli+hHD7azXlBTlYmNAJNKI+LyPtAD/AlEckF7L7UmAC1Hu+jtas/YjrkR7u4JJeLS9wdxJKfkYgIHLaO+YgV6PT1twIXAGtUdQDo4tRnSowx49h7xDfCK1I65L0QFxNFXlqCDR2OYIHeoQAsx/c8iv85vwtyPMbMSLsbOgAoy0/zOJLwVmjPokS0gBKKiPwe3zMe24GR6UAVSyjGBKSivoN5aQlkp8R7HUpYK8xK4uVKe+A4UgV6h7IGKNWprAtqjDmhor6dUrs7mdSCrCQaO/roHRgKuyWSzeQCHTa8Czj9GeCMmYV6B4bY39xlzV0BKMxKBLAn5iNUoHcoOcBuEXkT6BspVNWrxj/FGAO+RaiGhtUSSgBGnkU5fLSbJREwiaY5WaAJ5XY3gzBmJquo900BX5af7nEk4e/Esyht1jEfiQJKKKr6oogsBEpU9RkRSQKsgdOYAFTUd5CWEMP8zESvQwl7uanxxMdE2UivCBXoXF5/g2+9kn91igqAP7sVlDEzya66dsry0yNqyhWviAiFWUk2/UqECrRT/mbgQqADQFUrgdNbRNqYWaB3YIjd9R2sXpDhdSgRozAzkcNHrVM+EgWaUPqcVRcBcB5utCHExkxiZ107g8PK6gVjrvdmxjDycKM9pRB5Ak0oL4rIt4FEEfko8DDwn+6FZczM8M7BNgC7Q5mCwswkOvsGae8Z8DoUM0WBJpRbgWZgJ/BFfDMI/4NbQRkzU7xzqI2F2Unk2BPyASs8MeuwNXtFmkBHeQ2LyJ+BP6uqzYtgTABUlXcOHePCxTYV+1SMPNx4uK2blfNtqHUkmfAORXxuF5EWYC+w11mt8bsTnWeMgbpjPTR39nH2Qus/mYpCv3VRTGSZrMnra/hGd61V1SxVzQLOBS4UEVsR0ZgJvHXgKABnW4f8lKQlxJKeGGtDhyPQZAnleuA6Va0ZKVDVauAzwH93MzBjIt1rVa2kJ8ZSmmdTrkxVYVaiLbQVgSZLKLGq2jK60OlHiXUnJGMin6ry2v5Wzl+UTVSUPdA4VQuykqi1O5SIM1lC6Z/mPmNmtUNHu6k71sOFS6xDfjoKM5OobetheNieRYkkk43yOktEOsYoFyDBhXiMmRFerWoF4PzFOR5HEpnmZyXRPzRMU2cf89Ltv5pIMWFCUVWbANKYaXh+bxP56Qkszk32OpSIVOhMpHnoaLcllAgS6IONxpgA9Q4M8UplC+vPmGsTQk6TDR2OTJZQjAmy1/e30jMwxPozbP7U6SrISETE1kWJNJZQjAmyZ/Y0khQXzXmLrEN+uhJio5mbmmDTr0QYSyjGBNHg0DBP7DrCZcvmkBBrXZCnw/csit2hRBJXE4qIbBCRvSJSJSK3jrE/XkQedPZvE5Eiv323OeV7ReRyv/L7RKRJRHaNqitLRJ4WkUrnT3s82YTcq/tbae3q56pV+V6HEvEKM5OsDyXCuJZQRCQauAe4AigFrhOR0lGH3Qi0qeoS4G7gLufcUmATUAZsAH7u1AfwW6dstFuBZ1W1BHjWeW9MSD32bh2pCTFcuizX61Ai3vysJI509NI3OOR1KCZAbt6hrAOqVLXaWZxrM7Bx1DEbgQec7UeA9eIbFrMR2Kyqfc60L1VOfajqS8DRMa7nX9cDwNXB/DDGTKazd4AnKo5w5Yo84mOsuet0FWYmogr1x3q9DsUEyM2EUgAc9ntf65SNeYyqDgLtQHaA5442V1UbnO0jwNyxDhKRm0SkXETKm5ttJn4TPH9+t47u/iGuO3eB16HMCAts6HDEmZGd8upbO3TMORtU9V5VXaOqa3JzrVnCBIeq8m9vHGJlQTpn2RoeQXHiWRTrmI8YbiaUOqDQ7/18p2zMY5x16tOB1gDPHa1RRPKcuvKApmlHbswUvVrVyt7GTq4/b6E9zBgkc9MSiI0Wm8Y+griZUN4CSkSkWETi8HWybxl1zBbgBmf708Bzzt3FFmCTMwqsGCgB3pzkev513QA8FoTPYExAfvZcJfPSEti42kZ3BUt0lFCQkUitPYsSMVxLKE6fyC3Ak8Ae4CFVrRCRO0TkKuew3wDZIlIFfB1nZJaqVgAPAbuBJ4CbVXUIQET+CLwOLBORWhG50anrh8BHRaQS+Ijz3hjXbatu5c2ao3zxkkXWGR9khVlJ1uQVQQJaU366VHUrsHVU2Xf9tnuBa8Y5907gzjHKrxvn+FZg/enEa8xUqSr/9PQ+clLi2LTWOuODrTAriV07GyY/0ISFGdkpb0yo/NeuI7xZc5SvfmQpiXF2dxJshZlJtHUP0Nk74HUoJgCWUIyZpt6BIX6wdQ/L56WyaW3h5CeYKSvM8k1jb3N6RQZLKMZM0y9f3E9tWw/f/UQpMdH2o+SGwkwbOhxJ7KfAmGnY09DBPc9XcdVZ+VywxFZldIutixJZLKEYM0UDQ8P8j4d3kJ4Yy/euKvM6nBktMymWlPgYSygRwtVRXsbMRL98YT8V9R388jPnkJkc53U4M5qIsDA7iQOtllAigd2hGDMFexo6+NlzlXzyrHw2rJjndTizQlFOMgdau7wOwwTAEooxARoYGuabj1hTV6gVZydz+Gg3/YPDXodiJmEJxZgA/fKF/eyq6+D7G1eQZU1dIVOck8yw2kivSGAJxZgA7K7/oKnripV5XoczqxTlJANwoMWavcKdJRRjJtE/ODKqK447rKkr5IqdhFJjCSXs2SgvYyZxz/NV7G7o4N7rbVSXFzKTYklPjLWO+QhgdyjGTGBXXTv3PF/FX6wu4GNlNqrLCyJCUU6y3aFEAEsoxoxj5AHGrOQ4/vGT1tTlpeLsJA60WKd8uLOEYsw47n+1hvePdHLHxhWkJ8V6Hc6sVpSTTH17D70DQ16HYiZgCcWYMdS2dXP305V85Iw5XF421+twZr3inGRUseWAw5wlFGNGUVVu31IBwO1Xldka8WHARnpFBksoxozyZEUjz+xp4msfLWG+M3268dZIQtnffNzjSMxELKEY46d3YIjvP76b5fNS+dyFxV6HYxypCbHMS0ugqtESSjizhGKMn3974yB1x3r4X58oJdYWzQorJXNTqGyyhBLO7CfGGEdH7wD3PF/FxSU5XGiLZoWdkjmpVDUdZ3hYvQ7FjMMSijGOX71UTVv3AN/asNzrUMwYSuam0DMwRN0xW18+XFlCMQZo6uzl1y/X8Ikz81hRkO51OGYMJXNSAKhs6vQ4EjMeSyjGAP/vuSoGhob5xseWeR2KGceSkYRiHfNhyxKKmfUOtnbx79sOce3awhPDU034yUiKIzc13jrmw5glFDPr/fTpfcREC19ZX+J1KGYSS+emUNloTV7hytWEIiIbRGSviFSJyK1j7I8XkQed/dtEpMhv321O+V4RuXyyOkXktyJSIyLbndcqNz+bmRkq6tt5bHs9n7+wmDlpCV6HYyZRMieVShvpFbZcS04dmVYAABHaSURBVCgiEg3cA1wBlALXiUjpqMNuBNpUdQlwN3CXc24psAkoAzYAPxeR6ADq/KaqrnJe2936bGbm+NETe0lPjOWLlyz2OhQTgNK8NLr7h2xtlDDl5h3KOqBKVatVtR/YDGwcdcxG4AFn+xFgvfgmTtoIbFbVPlWtAaqc+gKp05iAvL6/lRf3NXPzZYtJT7TZhCNBaX4aABX1HR5HYsbiZkIpAA77va91ysY8RlUHgXYge4JzJ6vzThF5T0TuFpH4sYISkZtEpFxEypubm6f+qcyMoKrc9cT75KUn8N/PL/I6HBOgkrkpxESJJZQwNZM65W8DlgNrgSzgW2MdpKr3quoaVV2Tm5sbyvhMGHmyopHth4/x1Y+UkBAb7XU4JkDxMdGUzE2lor7d61DMGNxMKHVAod/7+U7ZmMeISAyQDrROcO64dapqg/r0Affjax4z5hSDQ8P85Km9LM5N5i/Pnu91OGaKyvLT2F3fgap1zIcbNxPKW0CJiBSLSBy+TvYto47ZAtzgbH8aeE59/0q2AJucUWDFQAnw5kR1ikie86cAVwO7XPxsJoI9+k4dVU3H+ebly4ixCSAjTll+Gq1d/TR19nkdihklxq2KVXVQRG4BngSigftUtUJE7gDKVXUL8Bvg9yJSBRzFlyBwjnsI2A0MAjer6hDAWHU6l/yDiOQCAmwH/tatz2YiV+/AEHc/s49VhRlcXjbP63DMNJTl+6bGqahvZ64N9Q4rriUUAFXdCmwdVfZdv+1e4Jpxzr0TuDOQOp3yD59uvGbm+/3rB2lo7+Wnf7XKVmKMUGfkpSICO2s7+PByW545nNj9vpk1OnoHuOeFKj60NJfzF2d7HY6ZptSEWJbOSeXtQ21eh2JGsYRiZo1/fXE/x7oH+J+X2wSQke6cokzePdhmT8yHGUsoZlY40t7Lfa8c4Kqz8m16+hngnAWZdPYN2kSRYcYSipkVfvr0XoaGlW/a3cmMcM7CTADePmjNXuHEEoqZ8fY0dPDw27XccMFCCrOSvA7HBMHC7CSyk+MoP3jU61CMH0soZsb7wdY9pCXEcstlNj39TCEinL0w0+5QwowlFDOjPf9+Ey9XtvD360tIT7IJIGeSdUVZHGztpqHd1pgPF5ZQzIzV0z/E/3psF4tzk7n+vIVeh2OC7KKSHABermzxOBIzwhKKmbF+9lwltW09/OBTK4mLsX/qM83yeankpMTziiWUsGE/ZWZG2lXXzq9equaac+Zz7iJ7iHEmEhEuLsnh1aoWex4lTFhCMTNOd/8gf//Hd8lJiefbV57hdTjGRReX5NDa1W/ro4QJSyhmxvnelt3UtHbx02vPIjM5zutwjIsuXTaH6CjhiYoGr0MxWEIxM8zv3zjIg+WH+dIli7lgcY7X4RiXZSXHcW5xFv+164itjxIGLKGYGeP5vU3cvqWCDy+fwzc+Zk/EzxZXrJhHdXOXTcMSBiyhmBnhpX3NfPH3b7Nsbio/u2410VE2Nf1scXnZPKIEtmyv9zqUWc8Siol4//FuLV/4XTmLc1P4wxfOJSXe1WV+TJiZk5bAJUtzefjtwwwODXsdzqxmCcVErL7BIX6wdQ9fe3AHZy/I4N+/cK51ws9S165dQGNHHy/ua/Y6lFnNfpUzEWlnbTv/4+Ed7G3s5DPnLeAfP1lGrK0PP2utP2MOOSnxPPD6QdafYas4esUSiokodcd6uPvpffzpnVrmpMZz32fX2DKwhtjoKD53YRE/fnIvO2vbWTnf1rzxgv1KZyLC4aPdfO8/K7jsJy+wZUc9f3PxIp766iWWTMwJ15+/kNSEGP7luUqvQ5m17A7FhC1VZUdtO79+uZqtOxuIEuHq1QV87aNLKchI9Do8E2bSEmL5wkWLuPuZfby2v8WeQ/KAJRQTdo519/Pnd+t4sLyWPQ0dpMbH8DcXL+KzFxaRl26JxIzvi5cs4uG3D/OPj1Xw+N9fRHxMtNchzSqWUExYON43yHPvN/HErgae2dNE/+AwKwrS+P7GMq5eXUBqgq1lYiaXEBvN9zeu4HO/fYv/s/V9br+qzOuQZhVLKMYTqsqho928WtXKs3saebmyhf6hYXJS4rlubSF/tbaQsnzrWDVTd9nyOdx4UTG/eaWGM/JSuXbtAq9DmjUsoZiQaero5bX9rbxa1cJr+1upO+Zbaa8gI5Hrz1/IhhXzOHtBpj3lbk7btzYsZ19jJ7c+uhPAkkqIWEIxrmlo76H8QBtvHTjKa/tbqXLmWkpPjOX8Rdn87SWLuGBJDotykhGxJGKCJy4minuvX8NNvy/nW3/ayfbDx7h1wxm2DLTLXE0oIrIB+GcgGvi1qv5w1P544HfAOUArcK2qHnD23QbcCAwBf6+qT05Up4gUA5uBbOBt4HpV7Xfz85kP9A4MsfdIJ+/VtVN+4CjlB9pO3IEkxUWztiiLa86Zz4VLcijNSyPK7kKMyxLjorn/s2v58VN7+dVL1TxV0chnLyji2nWFzElN8Dq8GUncmvJZRKKBfcBHgVrgLeA6Vd3td8zfAWeq6t+KyCbgU6p6rYiUAn8E1gH5wDPAUue0MesUkYeAR1V1s4j8Etihqr+YKMY1a9ZoeXl5ED/1zDQ4NExn7yCdvYN09A7Q1NlLXVsPtcd6qD3aw/tHOqhp6WJk0bw5qfGsLcpiTVEma4uyWD4vlRh7it14qKK+nbue2MtLztQsZ85PZ21RFsvmpVKck0xuSjw5qfEkx0Xb3XIARORtVV0zutzNO5R1QJWqVjsBbAY2Arv9jtkI3O5sPwL8P/H9bW4ENqtqH1AjIlVOfYxVp4jsAT4M/DfnmAeceidMKNP1s2cr2bKj/sT6CydSsp70x0nrM3xQNvJeT34/Kq+Prnv0eSeXja5Dx9k/dp1j7RvZGBxWegaGGEtstJCfkcjSual8/Mx8SvNSKctPZ35mov1QmrBSlp/O7z6/jv3Nx3li1xGef7+JP2w7SO/AqZNJxsdEkRAbTUJsFDFRvl+ERJwX4vzpW4JYAPzeR5IffGol64qzglqnmwmlADjs974WOHe8Y1R1UETa8TVZFQBvjDq3wNkeq85s4JiqDo5x/ElE5CbgJoAFC6bXUTcnNZ5lc1OdCk/648Q/qg/e+117nGM+qENOOmd0HfLBGR+UjapksnNP3X/qD4H/OdFRkBIfS2pCjPOKJTc1nvmZieSmxFvTlYkoi3NTuPmyJdx82RKGhn0jDWvbumnu7KO5s4+u/iH6BoboHRiid2CYgeHhk35RVFXnz5Pfo+NeMmwlxwf/GZ1Z1ymvqvcC94KvyWs6dWxat4BN62zUiDGRLDpKKM5Jpjgn2etQZgw3G7brgEK/9/OdsjGPEZEYIB1f5/x4545X3gpkOHWMdy1jjDEucjOhvAWUiEixiMQBm4Ato47ZAtzgbH8aeE59jflbgE0iEu+M3ioB3hyvTuec5506cOp8zMXPZowxZhTXmrycPpFbgCfxDfG9T1UrROQOoFxVtwC/AX7vdLofxZcgcI57CF8H/iBws6oOAYxVp3PJbwGbReR/A+86dRtjjAkR14YNRwIbNmyMMVM33rBhezjAGGNMUFhCMcYYExSWUIwxxgSFJRRjjDFBMas75UWkGTjo0eVzgBaPrj0Ri2tqLK6psbimJlzjWqiquaMLZ3VC8ZKIlI81SsJrFtfUWFxTY3FNTbjGNR5r8jLGGBMUllCMMcYEhSUU79zrdQDjsLimxuKaGotrasI1rjFZH4oxxpigsDsUY4wxQWEJxRhjTFBYQgkBEVklIm+IyHYRKReRdU65iMjPRKRKRN4TkbP9zrlBRCqd1w3j137asX1ZRN4XkQoR+ZFf+W1OXHtF5HK/8g1OWZWI3OpWXM61viEiKiI5zntPvy8R+bHzXb0nIv8hIhl++zz/vry8pnPdQhF5XkR2O/+evuKUZ4nI087fzdMikumUj/v36VJ80SLyrog87rwvFpFtzvUfdJbEwFk240GnfJuIFLkYU4aIPOL8u9ojIueHy/c1LapqL5dfwFPAFc72lcALftv/hW9F3vOAbU55FlDt/JnpbGe6ENdlwDNAvPN+jvNnKbADiAeKgf34lguIdrYXAXHOMaUufWeF+JYpOAjkhMn39TEgxtm+C7grXL4vvxhDfk2/a+cBZzvbqcA+57v5EXCrU36r3/c25t+ni/F9Hfh34HHn/UPAJmf7l8CXnO2/A37pbG8CHnQxpgeALzjbcUBGuHxf03nZHUpoKJDmbKcD9c72RuB36vMGvlUn84DLgadV9aiqtgFPAxtciOtLwA9VtQ9AVZv84tqsqn2qWgNUAeucV5WqVqtqP7DZOdYNdwP/k5NX6/b0+1LVp1R10Hn7Br6VQUfi8vr7GuHFNQFQ1QZVfcfZ7gT2AAXO9R9wDnsAuNrZHu/vM+hEZD7wceDXznsBPgw8Mk5cI/E+Aqx3jg92TOnAh3DWblLVflU9Rhh8X9NlCSU0vgr8WEQOAz8BbnPKC4DDfsfVOmXjlQfbUuBi57b+RRFZGw5xichGoE5Vd4za5fX35e/z+H5bDLe4vLjmKZxmotXANmCuqjY4u44Ac53tUMb6f/H9gjLsvM8Gjvn9guB/7RNxOfvbneODrRhoBu53muJ+LSLJhMf3NS2urdg424jIM8C8MXZ9B1gPfE1V/yQif4XvN5KPhEFcMfiaic4D1gIPiciiMIjr2/ial0JuorhU9THnmO/gW0n0D6GMLVKISArwJ+Crqtrh/8u9qqqIhPRZBRH5BNCkqm+LyKWhvPYkYoCzgS+r6jYR+Wd8TVwnePF9nQ5LKEGiquMmCBH5HfAV5+3DOLfdQB2+voIR852yOuDSUeUvuBDXl4BH1ddA+6aIDOObjG68uJigPChxichKfL+57XD+I5oPvCO+gQyefl9OfJ8FPgGsd743JoiLCcrdMlEsrhORWHzJ5A+q+qhT3Cgieara4DTRjDSthirWC4GrRORKIAFf8/M/42syinHuQvyvPRJXrYjE4GumbnUhrlqgVlW3Oe8fwZdQvP6+ps/rTpzZ8MLXlnyps70eeNvZ/jgnd7K96ZRnATX4Opgzne0sF+L6W+AOZ3spvttpAco4uZO5Gl9nb4yzXcwHHb5lLn93B/igU97r72sDsBvIHVUeTt9XyK/pd20Bfgf831HlP+bkTuYfTfT36XKMl/JBp/zDnNwp/3fO9s2c3Cn/kIvxvAwsc7Zvd76rsPm+pvx5vA5gNryAi4C3nR/ubcA5TrkA9+AblbMTWON3zufxde5WAZ9zKa444N+AXcA7wIf99n3HiWsvzgg1p/xKfKN39uNrBnL7u/NPKF5/X1X4ku525/XLcPu+vLqmc92L8A2ieM/vO7oSX//Ds0AlvlGFWZP9fboYo39CWQS86fy9PswHox0TnPdVzv5FLsazCih3vrM/4/uFKGy+r6m+bOoVY4wxQWGjvIwxxgSFJRRjjDFBYQnFGGNMUFhCMcYYExSWUIwxxgSFJRRjQkBEvuPMwPue+GadPtfrmIwJNntS3hiXicj5+J6uP1tV+8Q3HX/cadQ38nS3MWHF7lCMcV8e0KIfzOrcoqr1IrJWRF4TkR0i8qaIpIpIgojcLyI7nQkDLwPflC8iskVEngOeFZFkEbnPOe9dZ0JNYzxldyjGuO8p4Lsisg/fk88PAq87f16rqm+JSBrQg2/ON1XVlSKyHHhKRJY69ZwNnKmqR0XkB8Bzqvp58S309aaIPKOqXaH+cMaMsDsUY1ymqseBc4Cb8E1X/iDwRaBBVd9yjulwmrEuwjcdDqr6Pr4FxkYSytOqetTZ/hhwq4hsxzcRZgKwICQfyJhx2B2KMSGgqkP4/uN/QUR24puAcKr87z4E+EtV3RuE8IwJCrtDMcZlIrJMREr8ilbhm4E6b2RRM6f/JAbf7LN/7ZQtxXfXMVbSeBL48shKgiKy2sWPYExA7A7FGPelAP/i9HUM4pvF9ibgfqc8EV//yUeAnwO/cO5iBoHPOiPDRtf5fXyrEL4nIlH4puz/RCg+jDHjsdmGjTHGBIU1eRljjAkKSyjGGGOCwhKKMcaYoLCEYowxJigsoRhjjAkKSyjGGGOCwhKKMcaYoPj/AVNT7fllEc9eAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kGn1dFk2c5mV",
        "colab_type": "text"
      },
      "source": [
        "## Where to go from here?\n",
        "\n",
        "I am a bit frustrated by lack of stability that I am seeing in my implmentation of the Deep Q algorithm: sometimes the algorithm converges and sometimes not. Perhaps more tuning of hyper-parameters or use of a different optimization algorithm would exhibit better convergence. I have already spent more time than I had allocated on playing around with this agorithm so I am not going to try and fine-tune the hyperparamters or explore alternative optimization algorithms for now.\n",
        "\n",
        "Rather than spending time tuning hyperparameters I think it would be better use of my time to explore algorithmic improvements. In future posts I plan to cover the following extensions of the DQN algorithm: [Double Q-Learning](https://arxiv.org/abs/1509.06461), [Prioritized Experience Replay](https://arxiv.org/abs/1509.06461), and [Dueling Network Architectures](https://arxiv.org/abs/1511.06581)"
      ]
    }
  ]
}